Many thanks Russell. That worked
val *HiveDF* = Try(spark.read.
format("jdbc").
option("url", jdbcUrl).
option("dbtable", HiveSchema+"."+HiveTable).
option("user", HybridServerUserName).
option("password", HybridServerPassword).
load()) match {
* case Success(df) => df*
case Failure(e) => throw new Exception("Error
Encountered reading Hive table")
}
HiveDF: org.apache.spark.sql.DataFrame = [id: int, clustered: int ... 5
more fields]
Appreciated your help Sean and Russell
Mich
On Fri, 2 Oct 2020 at 01:22, Russell Spitzer <[email protected]>
wrote:
> You can't use df as the name of the return from the try and the name of
> the match variable in success. You also probably want to match the name of
> the variable in the match with the return from the match.
>
> So
>
> val df = Try(spark.read.
>
> format("jdbc").
>
> option("url", jdbcUrl).
>
> option("dbtable", HiveSchema+"."+HiveTable).
>
> option("user", HybridServerUserName).
>
> option("password", HybridServerPassword).
>
> load()) match {
>
> * case Success(validDf) => validDf*
>
> case Failure(e) => throw new Exception("Error
> Encountered reading Hive table")
>
> }
>
> On Thu, Oct 1, 2020 at 5:53 PM Mich Talebzadeh <[email protected]>
> wrote:
>
>>
>> Many thanks SEan.
>>
>>
>> Maybe I misunderstood your point?
>>
>>
>> var DF = Try(spark.read.
>>
>> format("jdbc").
>>
>> option("url", jdbcUrl).
>>
>> option("dbtable", HiveSchema+"."+HiveTable).
>>
>> option("user", HybridServerUserName).
>>
>> option("password", HybridServerPassword).
>>
>> load()) match {
>>
>> * case Success(DF) => HiveDF*
>>
>> case Failure(e) => throw new Exception("Error
>> Encountered reading Hive table")
>>
>> }
>>
>> Still getting the error
>>
>>
>> <console>:74: error: recursive method DF needs type
>>
>> case Success(DF) => HiveDF
>>
>> Do I need to define DF as DataFrame beforehand because at that moment
>> Spark does not know what DF type is
>>
>> Thanks again
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Thu, 1 Oct 2020 at 23:08, Sean Owen <[email protected]> wrote:
>>
>>> You are reusing HiveDF for two vars and it ends up ambiguous. Just
>>> rename one.
>>>
>>> On Thu, Oct 1, 2020, 5:02 PM Mich Talebzadeh <[email protected]>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>> Spark version 2.3.3 on Google Dataproc
>>>>
>>>>
>>>> I am trying to use databricks to other databases
>>>>
>>>>
>>>> https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html
>>>>
>>>>
>>>> to read from Hive table on Prem using Spark in Cloud
>>>>
>>>>
>>>> This works OK without a Try enclosure.
>>>>
>>>>
>>>> import spark.implicits._
>>>>
>>>> import scala.util.{Try, Success, Failure}
>>>>
>>>> val HiveDF = Try(spark.read.
>>>>
>>>> format("jdbc").
>>>>
>>>> option("url", jdbcUrl).
>>>>
>>>> option("dbtable", HiveSchema+"."+HiveTable).
>>>>
>>>> option("user", HybridServerUserName).
>>>>
>>>> option("password", HybridServerPassword).
>>>>
>>>> load()) match {
>>>>
>>>> case Success(HiveDF) => HiveDF
>>>>
>>>> case Failure(e) => throw new Exception("Error
>>>> Encountered reading Hive table")
>>>>
>>>> }
>>>>
>>>> However, with Try I am getting the following error
>>>>
>>>>
>>>> <console>:66: error: recursive value HiveDF needs type
>>>>
>>>> case Success(HiveDF) => HiveDF
>>>>
>>>> Wondering what is causing this. I have used it before (say reading from
>>>> an XML file) and it worked the,
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>
