Is there a reason why different Scala (it seems at least 2.10/2.11) versions
are mixed? This never works.
Do you include by accident a dependency to with an old Scala version? Ie the
Hbase datasource maybe?
> Am 17.02.2020 um 22:15 schrieb Mich Talebzadeh <mich.talebza...@gmail.com>:
>
>
> Thanks Muthu,
>
>
> I am using the following jar files for now in local mode i.e.
> spark-shell_local --jars …..
>
> json4s-jackson_2.10-3.2.10.jar
> json4s_2.11-3.2.11.jar
> json4s-native_2.10-3.4.0.jar
>
> Which one is the incorrect one please/
>
> Regards,
>
> Mich
>
>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss,
> damage or destruction of data or any other property which may arise from
> relying on this email's technical content is explicitly disclaimed. The
> author will in no case be liable for any monetary damages arising from such
> loss, damage or destruction.
>
>
>
>> On Mon, 17 Feb 2020 at 20:28, Muthu Jayakumar <bablo...@gmail.com> wrote:
>> I suspect the spark job is somehow having an incorrect (newer) version of
>> json4s in the classpath. json4s 3.5.3 is the utmost version that can be
>> used.
>>
>> Thanks,
>> Muthu
>>
>>> On Mon, Feb 17, 2020, 06:43 Mich Talebzadeh <mich.talebza...@gmail.com>
>>> wrote:
>>> Hi,
>>>
>>> Spark version 2.4.3
>>> Hbase 1.2.7
>>>
>>> Data is stored in Hbase as Json. example of a row shown below
>>> <image.png>
>>>
>>> I am trying to read this table in Spark Scala
>>>
>>> import org.apache.spark.sql.{SQLContext, _}
>>> import org.apache.spark.sql.execution.datasources.hbase._
>>> import org.apache.spark.{SparkConf, SparkContext}
>>> import spark.sqlContext.implicits._
>>> import org.json4s._
>>> import org.json4s.jackson.JsonMethods._
>>> import org.json4s.jackson.Serialization.{read => JsonRead}
>>> import org.json4s.jackson.Serialization.{read, write}
>>> def catalog = s"""{
>>> | "table":{"namespace":"trading", "name":"MARKETDATAHBASEBATCH",
>>> | "rowkey":"key",
>>> | "columns":{
>>> | "rowkey":{"cf":"rowkey", "col":"key", "type":"string"},
>>> | |"ticker":{"cf":"PRICE_INFO", "col":"ticker", "type":"string"},
>>> | |"timeissued":{"cf":"PRICE_INFO", "col":"timeissued",
>>> "type":"string"},
>>> | |"price":{"cf":"PRICE_INFO", "col":"price", "type":"string"}
>>> | |}
>>> | |}""".stripMargin
>>> def withCatalog(cat: String): DataFrame = {
>>> spark.sqlContext
>>> .read
>>> .options(Map(HBaseTableCatalog.tableCatalog->cat))
>>> .format("org.apache.spark.sql.execution.datasources.hbase")
>>> .load()
>>> }
>>> val df = withCatalog(catalog)
>>>
>>>
>>> However, I am getting this error
>>>
>>> Spark session available as 'spark'.
>>> java.lang.NoSuchMethodError:
>>> org.json4s.jackson.JsonMethods$.parse(Lorg/json4s/JsonInput;Z)Lorg/json4s/JsonAST$JValue;
>>> at
>>> org.apache.spark.sql.execution.datasources.hbase.HBaseTableCatalog$.apply(HBaseTableCatalog.scala:257)
>>> at
>>> org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.<init>(HBaseRelation.scala:80)
>>> at
>>> org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(HBaseRelation.scala:51)
>>> at
>>> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
>>> at
>>> org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
>>> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
>>> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
>>> at withCatalog(testme.scala:49)
>>> ... 65 elided
>>>
>>> I have Googled it but with little luck!
>>>
>>> Thanks,
>>> Mich
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>> Disclaimer: Use it at your own risk. Any and all responsibility for any
>>> loss, damage or destruction of data or any other property which may arise
>>> from relying on this email's technical content is explicitly disclaimed.
>>> The author will in no case be liable for any monetary damages arising from
>>> such loss, damage or destruction.
>>>
