Hi Ricardo,
I tried that also
still that same error
Thanks,
Divya
On 11 April 2016 at 18:51, Ricardo Crespo <ricardo.e.cre...@gmail.com>
wrote:
> Yes sorry, I didnt see it at first.
>
>
> Could you test this:
>
>
> Instead of:
>
> --conf
> "spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-client.jar"
>
> using it without quotes:
>
> --conf
> spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-client.jar
>
>
>
>
> 2016-04-11 12:43 GMT+02:00 Divya Gehlot <divya.htco...@gmail.com>:
>
>> Hi Ricardo,
>> Are you talking about below Phoenix jar ?
>> --conf "spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/
>> phoenix-client.jar"
>>
>> That also been added.
>> Please refer to my first post .
>>
>> Thanks,
>> Divya
>>
>> On 11 April 2016 at 18:33, Ricardo Crespo <ricardo.e.cre...@gmail.com>
>> wrote:
>>
>>> Divya,
>>>
>>>
>>> I check that the error that you are getting is on the executor, could
>>> you add this parameter to spark submit:
>>>
>>>
>>> --conf spark.executor.extraClassPath=path_to_phoenix_jar
>>>
>>>
>>> That way the executor should get phoenix in the classpath.
>>>
>>>
>>> Best Regards,
>>> Ricardo
>>>
>>>
>>> 2016-04-11 12:15 GMT+02:00 Divya Gehlot <divya.htco...@gmail.com>:
>>>
>>>> Hi Ricardo ,
>>>> If would have observed my previous post carefully
>>>> I am already passing the below jars
>>>> --jars /usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-spark-4.4.0.2.3.4.
>>>> 0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/phoenix-client.
>>>> jar,/usr/hdp/2.3.4.0-3485/phoenix/phoenix-server.jar
>>>>
>>>> When I try to test the connection through JDBC able to connect
>>>> succssfully
>>>> /* Testing Phoenix Connection */
>>>> import java.sql.{Connection, DriverManager}
>>>> def testConnection(): Unit = {
>>>> Class.forName("org.apache.phoenix.jdbc.PhoenixDriver")
>>>> println("start to connect phoenix!")
>>>> var conn =
>>>> DriverManager.getConnection("jdbc:phoenix:localhost:2181")
>>>> println("Connected to Phoenix")
>>>> }
>>>>
>>>>
>>>> StackTrace of above code
>>>>
>>>>
>>>> import java.sql.{Connection, DriverManager}
>>>> testConnection: ()Unit
>>>> start to connect phoenix!
>>>> 16/04/11 10:04:51 INFO RecoverableZooKeeper: Process
>>>> identifier=hconnection-0x4260f924 connecting to ZooKeeper
>>>> ensemble=localhost:2181
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client
>>>> environment:zookeeper.version=3.4.6-3485--1, built on 12/16/2015 02:35 GMT
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client environment:host.name
>>>> =ip-xxx-xx-xx-xxx.ap-southeast-1.compute.internal
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client
>>>> environment:java.version=1.7.0_67
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client environment:java.vendor=Oracle
>>>> Corporation
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client
>>>> environment:java.home=/usr/jdk64/jdk1.7.0_67/jre
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client
>>>> environment:java.class.path=/usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-core-4.4.0.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-spark-4.4.0.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar:/usr/hdp/current/spark-thriftserver/conf/:/usr/hdp/2.3.4.0-3485/spark/lib/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar:/usr/hdp/2.3.4.0-3485/spark/lib/datanucleus-api-jdo-3.2.6.jar:/usr/hdp/2.3.4.0-3485/spark/lib/datanucleus-core-3.2.10.jar:/usr/hdp/2.3.4.0-3485/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client
>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client environment:java.compiler=<NA>
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client environment:os.name=Linux
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client environment:os.arch=amd64
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client
>>>> environment:os.version=3.10.0-229.el7.x86_64
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client environment:user.name=hdfs
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client
>>>> environment:user.home=/home/hdfs
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Client environment:user.dir=/home/hdfs
>>>> 16/04/11 10:04:51 INFO ZooKeeper: Initiating client connection,
>>>> connectString=localhost:2181 sessionTimeout=90000
>>>> watcher=hconnection-0x4260f9240x0, quorum=localhost:2181,
>>>> baseZNode=/hbase-unsecure
>>>> 16/04/11 10:04:51 INFO ClientCnxn: Opening socket connection to server
>>>> localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL
>>>> (unknown error)
>>>> 16/04/11 10:04:51 INFO ClientCnxn: Socket connection established to
>>>> localhost/127.0.0.1:2181, initiating session
>>>> 16/04/11 10:04:51 INFO ClientCnxn: Session establishment complete on
>>>> server localhost/127.0.0.1:2181, sessionid = 0x353f97f0405053a,
>>>> negotiated timeout = 40000
>>>> 16/04/11 10:04:51 INFO Metrics: Initializing metrics system: phoenix
>>>> 16/04/11 10:04:51 INFO MetricsConfig: loaded properties from
>>>> hadoop-metrics2.properties
>>>> 16/04/11 10:04:51 INFO MetricsSystemImpl: Scheduled snapshot period at
>>>> 60 second(s).
>>>> 16/04/11 10:04:51 INFO MetricsSystemImpl: phoenix metrics system started
>>>> 16/04/11 10:04:52 INFO RecoverableZooKeeper: Process
>>>> identifier=hconnection-0x1679b42e connecting to ZooKeeper
>>>> ensemble=localhost:2181
>>>> 16/04/11 10:04:52 INFO ZooKeeper: Initiating client connection,
>>>> connectString=localhost:2181 sessionTimeout=90000
>>>> watcher=hconnection-0x1679b42e0x0, quorum=localhost:2181,
>>>> baseZNode=/hbase-unsecure
>>>> 16/04/11 10:04:52 INFO ClientCnxn: Opening socket connection to server
>>>> localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL
>>>> (unknown error)
>>>> 16/04/11 10:04:52 INFO ClientCnxn: Socket connection established to
>>>> localhost/127.0.0.1:2181, initiating session
>>>> 16/04/11 10:04:52 INFO ClientCnxn: Session establishment complete on
>>>> server localhost/127.0.0.1:2181, sessionid = 0x353f97f0405053b,
>>>> negotiated timeout = 40000
>>>> 16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation:
>>>> Closing master protocol: MasterService
>>>> 16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation:
>>>> Closing zookeeper sessionid=0x353f97f0405053b
>>>> 16/04/11 10:04:52 INFO ZooKeeper: Session: 0x353f97f0405053b closed
>>>> 16/04/11 10:04:52 INFO ClientCnxn: EventThread shut down
>>>> 16/04/11 10:04:52 INFO RecoverableZooKeeper: Process
>>>> identifier=hconnection-0x6c4546cf connecting to ZooKeeper
>>>> ensemble=localhost:2181
>>>> 16/04/11 10:04:52 INFO ZooKeeper: Initiating client connection,
>>>> connectString=localhost:2181 sessionTimeout=90000
>>>> watcher=hconnection-0x6c4546cf0x0, quorum=localhost:2181,
>>>> baseZNode=/hbase-unsecure
>>>> 16/04/11 10:04:52 INFO ClientCnxn: Opening socket connection to server
>>>> localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL
>>>> (unknown error)
>>>> 16/04/11 10:04:52 INFO ClientCnxn: Socket connection established to
>>>> localhost/0:0:0:0:0:0:0:1:2181, initiating session
>>>> 16/04/11 10:04:52 INFO ClientCnxn: Session establishment complete on
>>>> server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x353f97f0405053c,
>>>> negotiated timeout = 40000
>>>> 16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation:
>>>> Closing master protocol: MasterService
>>>> 16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation:
>>>> Closing zookeeper sessionid=0x353f97f0405053c
>>>> 16/04/11 10:04:52 INFO ZooKeeper: Session: 0x353f97f0405053c closed
>>>> 16/04/11 10:04:52 INFO ClientCnxn: EventThread shut down
>>>> 16/04/11 10:04:52 INFO RecoverableZooKeeper: Process
>>>> identifier=hconnection-0x5be1d976 connecting to ZooKeeper
>>>> ensemble=localhost:2181
>>>> 16/04/11 10:04:52 INFO ZooKeeper: Initiating client connection,
>>>> connectString=localhost:2181 sessionTimeout=90000
>>>> watcher=hconnection-0x5be1d9760x0, quorum=localhost:2181,
>>>> baseZNode=/hbase-unsecure
>>>> 16/04/11 10:04:52 INFO ClientCnxn: Opening socket connection to server
>>>> localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL
>>>> (unknown error)
>>>> 16/04/11 10:04:52 INFO ClientCnxn: Socket connection established to
>>>> localhost/0:0:0:0:0:0:0:1:2181, initiating session
>>>> 16/04/11 10:04:52 INFO ClientCnxn: Session establishment complete on
>>>> server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x353f97f0405053d,
>>>> negotiated timeout = 40000
>>>> 16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation:
>>>> Closing master protocol: MasterService
>>>> 16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation:
>>>> Closing zookeeper sessionid=0x353f97f0405053d
>>>> 16/04/11 10:04:52 INFO ZooKeeper: Session: 0x353f97f0405053d closed
>>>> 16/04/11 10:04:52 INFO ClientCnxn: EventThread shut down
>>>> 16/04/11 10:04:52 INFO RecoverableZooKeeper: Process
>>>> identifier=hconnection-0x46f4dedf connecting to ZooKeeper
>>>> ensemble=localhost:2181
>>>> 16/04/11 10:04:52 INFO ZooKeeper: Initiating client connection,
>>>> connectString=localhost:2181 sessionTimeout=90000
>>>> watcher=hconnection-0x46f4dedf0x0, quorum=localhost:2181,
>>>> baseZNode=/hbase-unsecure
>>>> 16/04/11 10:04:52 INFO ClientCnxn: Opening socket connection to server
>>>> localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL
>>>> (unknown error)
>>>> 16/04/11 10:04:52 INFO ClientCnxn: Socket connection established to
>>>> localhost/127.0.0.1:2181, initiating session
>>>> 16/04/11 10:04:52 INFO ClientCnxn: Session establishment complete on
>>>> server localhost/127.0.0.1:2181, sessionid = 0x353f97f0405053e,
>>>> negotiated timeout = 40000
>>>> 16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation:
>>>> Closing master protocol: MasterService
>>>> 16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation:
>>>> Closing zookeeper sessionid=0x353f97f0405053e
>>>> 16/04/11 10:04:52 INFO ZooKeeper: Session: 0x353f97f0405053e closed
>>>> 16/04/11 10:04:52 INFO ClientCnxn: EventThread shut down
>>>> Connected to Phoenix
>>>> calling testConnection()
>>>>
>>>>
>>>> Unable to figure out the root cause
>>>> Why its throwing error when I try to save to Phoenix
>>>> Would really appreciate the help.
>>>>
>>>>
>>>> Thanks,
>>>> Divya
>>>>
>>>> On 11 April 2016 at 17:33, Ricardo Crespo <ricardo.e.cre...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Divya,
>>>>>
>>>>>
>>>>> I am asumming you are using YARN, you have to put phoenix jar on your
>>>>> spark classpath, you can do it with this parameter:
>>>>>
>>>>> --jars path_to_phoenix_jar
>>>>>
>>>>> you can check another options here:
>>>>> http://spark.apache.org/docs/latest/running-on-yarn.html
>>>>>
>>>>>
>>>>> Another option is to included phoenix on your jar with mvn assembly
>>>>> plugin:
>>>>>
>>>>> http://maven.apache.org/plugins/maven-assembly-plugin/
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Ricardo
>>>>>
>>>>>
>>>>>
>>>>> 2016-04-11 11:17 GMT+02:00 Divya Gehlot <divya.htco...@gmail.com>:
>>>>>
>>>>>> Hi,
>>>>>> I am getting below error when I try to save data to Phoenix
>>>>>> Below are Cluster configuration and steps which I followed :
>>>>>> *Cluster Configuration :*
>>>>>> Hortonworks distribution 2.3.4 version
>>>>>> Spark 1.5.2
>>>>>> Pheonix 4.4
>>>>>>
>>>>>>
>>>>>> *Table created in Phoenix *
>>>>>> CREATE TABLE TEST
>>>>>> (
>>>>>> RPT_DATE varchar(100) PRIMARY KEY
>>>>>> )
>>>>>> ;
>>>>>>
>>>>>> *Spark Scala Script *
>>>>>> val dfLCR = readTable(sqlContext, "", "TEST")
>>>>>> val schemaL = dfLCR.schema
>>>>>> val lcrReportPath = "/TestDivya/Spark/Results/TestData/"
>>>>>> val dfReadReport=
>>>>>> sqlContext.read.format("com.databricks.spark.csv").option("header",
>>>>>> "true").schema(schemaL).load(lcrReportPath)
>>>>>> dfReadlcrReport.show()
>>>>>> val dfWidCol = dfReadReport.withColumn("RPT_DATE",lit("2015-01-01"))
>>>>>> val dfSelect = dfWidCol.select("RPT_DATE")
>>>>>>
>>>>>> dfSelect.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(collection.immutable.Map(
>>>>>> "zkUrl" -> "localhost",
>>>>>> "table" -> "TEST")).save()
>>>>>>
>>>>>> *command line to run the Spark scala job *
>>>>>> spark-shell --properties-file /TestDivya/Spark/Phoenix.properties
>>>>>> --jars
>>>>>> /usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-spark-4.4.0.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/phoenix-client.jar,/usr/hdp/2.3.4.0-3485/phoenix/phoenix-server.jar
>>>>>> --driver-class-path
>>>>>> /usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-spark-4.4.0.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
>>>>>> --conf
>>>>>> "spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-client.jar"
>>>>>> --packages com.databricks:spark-csv_2.10:1.4.0 --master yarn-client -i
>>>>>> /TestDivya/Spark/WriteToPheonix.scala
>>>>>>
>>>>>>
>>>>>> *Error :*
>>>>>>
>>>>>> 16/04/11 09:03:43 INFO TaskSetManager: Lost task 2.3 in stage 3.0
>>>>>> (TID 409) on executor ip-xxx-xx-xx-xxx.ap-southeast-1.compute.internal:
>>>>>> java.lang.RuntimeException (java.sql.SQLException: No suitable driver
>>>>>> found
>>>>>> for jdbc:phoenix:localhost:2181:/hbase-unsecure;) [duplicate 7]
>>>>>> org.apache.spark.SparkException: Job aborted due to stage failure:
>>>>>> Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in
>>>>>> stage 3.0 (TID 408, ip-172-31-22-135.ap-southeast-1.compute.internal):
>>>>>> java.lang.RuntimeException: java.sql.SQLException: No suitable driver
>>>>>> found
>>>>>> for jdbc:phoenix:localhost:2181:/hbase-unsecure;
>>>>>>
>>>>>> Could somebody help me resolving the error.
>>>>>> What am I missing ?
>>>>>>
>>>>>> Thanks,
>>>>>> Divya
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
