Re: Spark SQL JDBC teradata syntax error

What is the query On Fri, May 3, 2019 at 5:28 PM KhajaAsmath Mohammed wrote: > Hi > > I have followed link > https://community.teradata.com/t5/Connectivity/Teradata-JDBC-Driver-returns-the-wrong-schema-column-nullability/m-p/77824 > to > connect teradata from spark. > > I was able to print sche

Spark SQL JDBC teradata syntax error

Hi I have followed link https://community.teradata.com/t5/Connectivity/Teradata-JDBC-Driver-returns-the-wrong-schema-column-nullability/m-p/77824 to connect teradata from spark. I was able to print schema if I give table name instead of sql query. I am getting below error if I give query(code sn

[Spark SQL] JDBC connection from UDF

Hello guys, I'm using Spark SQL with Hive thru Thrift. I need this because I need to create a table by table mask. Here is an example: 1. Take tables by mask, like SHOW TABLES IN db 'table__*' 2. Create query like: CREATE TABLE total_data AS SELECT * FROM table__1 UNION ALL SELECT * FROM table__2

Spark SQL -JDBC connectivity

Hi, I would to know the steps to connect SPARK SQL from spring framework (Web-UI). also how to run and deploy the web application?

Re: spark-sql jdbc dataframe mysql data type issue

select 10 sample rows for columns id, ctime from each (MySQL and spark) tables and post the output please. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

spark-sql jdbc dataframe mysql data type issue

Hi, I came across this strange behavior of Apache Spark 1.6.1: when I was reading mysql table into spark dataframe ,a column of data type float got mapped into double. dataframe schema: root |-- id: long (nullable = true) |-- ctime: double (nullable = true) |-- atime: double (nullable = tru

Re: Having issue with Spark SQL JDBC on hive table !!!

gt; hive.server2.thrift.bind.host=myhost --hiveconf >>>>> hive.server2.thrift.port=* >>>>> >>>>> and also able to connect through beeline >>>>> >>>>> *beeline>* !connect jdbc:hive2://192.168.145.20: >>>&

Re: Having issue with Spark SQL JDBC on hive table !!!

ve.server2.thrift.port=* >>>> >>>> and also able to connect through beeline >>>> >>>> *beeline>* !connect jdbc:hive2://192.168.145.20:9999 >>>> Enter username for jdbc:hive2://192.168.145.20:: root >>>> Enter pass

Re: Having issue with Spark SQL JDBC on hive table !!!

Enter password for jdbc:hive2://192.168.145.20:9999: impetus >>> *beeline > * >>> >>> It is not giving query result on hive table through Spark JDBC, but it >>> is working with spark HiveSQLContext. See complete scenario explain below. >>> >>> Help me

Re: Having issue with Spark SQL JDBC on hive table !!!

.145.20:: root >> Enter password for jdbc:hive2://192.168.145.20:: impetus >> *beeline > * >> >> It is not giving query result on hive table through Spark JDBC, but it is >> working with spark HiveSQLContext. See complete scenario explain below. >> &g

Re: Having issue with Spark SQL JDBC on hive table !!!

eSQLContext. See complete scenario explain below. > > Help me understand the issue why Spark SQL JDBC is not giving result ? > > Below are version details. > > *Hive Version : 1.2.1* > *Hadoop Version : 2.6.0* > *Spark version: 1.3.1* > > Let me know if

Having issue with Spark SQL JDBC on hive table !!!

g with spark HiveSQLContext. See complete scenario explain below. Help me understand the issue why Spark SQL JDBC is not giving result ? Below are version details. *Hive Version : 1.2.1* *Hadoop Version : 2.6.0* *Spark version: 1.3.1* Let me know if need other details. *Created Hive

Re: Spark sql jdbc fails for Oracle NUMBER type columns

; , "user@spark.apache.org" > > > > Date: 11/05/2015 06:35 PM > > Subject: Re: Spark sql jdbc fails for Oracle NUMBER type columns > > > > Hi Richard, > > > Thank you for the updates. Do you know tentative timeline for 1.6 > > release? Mean whi

Re: Spark sql jdbc fails for Oracle NUMBER type columns

bhattula Rajesh Kumar > To: Richard Hillegas/San Francisco/IBM@IBMUS > Cc: "u...@spark.incubator.apache.org" > , "user@spark.apache.org" > > Date: 11/05/2015 06:35 PM > Subject: Re: Spark sql jdbc fails for Oracle NUMBER type columns > > Hi Richard, > Th

Re: Spark sql jdbc fails for Oracle NUMBER type columns

o/IBM@IBMUS > > To: Madabhattula Rajesh Kumar > > Cc: "user@spark.apache.org" , > > "u...@spark.incubator.apache.org" > > Date: 11/05/2015 09:17 AM > > Subject: Re: Spark sql jdbc fails for Oracle NUMBER type columns > > > > > Hi Rajesh

Re: Spark sql jdbc fails for Oracle NUMBER type columns

ard Hillegas/San Francisco/IBM@IBMUS > To: Madabhattula Rajesh Kumar > Cc: "user@spark.apache.org" , > "u...@spark.incubator.apache.org" > Date: 11/05/2015 09:17 AM > Subject: Re: Spark sql jdbc fails for Oracle NUMBER type columns > > Hi Rajesh, &g

Re: Spark sql jdbc fails for Oracle NUMBER type columns

they will have the spare cycles to vet that pull request. Thanks, Rick Madabhattula Rajesh Kumar wrote on 11/05/2015 05:51:29 AM: > From: Madabhattula Rajesh Kumar > To: "user@spark.apache.org" , > "u...@spark.incubator.apache.org" > Date: 11/05/2015 05:51 AM

Spark sql jdbc fails for Oracle NUMBER type columns

Hi, Is this issue fixed in 1.5.1 version? Regards, Rajesh

Re: databases currently supported by Spark SQL JDBC

I suppose every RDBMS has a jdbc driver to connct to. I know Oracle, MySQL, SQL Server, Terdata, Netezza have. On Thu, Jul 9, 2015 at 10:09 PM, Niranda Perera wrote: > Hi, > > I'm planning to use Spark SQL JDBC datasource provider in various RDBMS > databases. > >

databases currently supported by Spark SQL JDBC

Hi, I'm planning to use Spark SQL JDBC datasource provider in various RDBMS databases. what are the databases currently supported by Spark JDBC relation provider? rgds -- Niranda @n1r44 <https://twitter.com/N1R44> https://pythagoreanscript.wordpress.com/

Re: Spark SQL JDBC Source data skew

Can some one help me here? Please On Sat, Jun 20, 2015 at 9:54 AM Sathish Kumaran Vairavelu < vsathishkuma...@gmail.com> wrote: > Hi, > > In Spark SQL JDBC data source there is an option to specify upper/lower > bound and num of partitions. How Spark handles data distribution, i

Spark SQL JDBC Source data skew

Hi, In Spark SQL JDBC data source there is an option to specify upper/lower bound and num of partitions. How Spark handles data distribution, if we do not give the upper/lower/num of parititons ? Will all data from the external data source skewed up in one executor? In many situations, we do not

Re: Spark SQL JDBC Source Join Error

Thank you.. it works in Spark 1.4. On Sun, Jun 14, 2015 at 3:51 PM Michael Armbrust wrote: > Sounds like SPARK-5456 . > Which is fixed in Spark 1.4. > > On Sun, Jun 14, 2015 at 11:57 AM, Sathish Kumaran Vairavelu < > vsathishkuma...@gmail.com> wr

Re: Spark SQL JDBC Source Join Error

Sounds like SPARK-5456 . Which is fixed in Spark 1.4. On Sun, Jun 14, 2015 at 11:57 AM, Sathish Kumaran Vairavelu < vsathishkuma...@gmail.com> wrote: > Hello Everyone, > > I pulled 2 different tables from the JDBC source and then joined them > usi

Spark SQL JDBC Source Join Error

Hello Everyone, I pulled 2 different tables from the JDBC source and then joined them using the cust_id *decimal* column. A simple join like as below. This simple join works perfectly in the database but not in Spark SQL. I am importing 2 tables as a data frame/registertemptable and firing sql on

RE: Anybody using Spark SQL JDBC server with DSE Cassandra?

I am considering DSE, which has integrated Spark SQL Thrift/JDBC server with Cassandra. Mohammed From: Deenar Toraskar [mailto:deenar.toras...@gmail.com] Sent: Thursday, June 4, 2015 7:42 AM To: Mohammed Guller Cc: user@spark.apache.org Subject: Re: Anybody using Spark SQL JDBC server with DSE

Re: Anybody using Spark SQL JDBC server with DSE Cassandra?

Mohammed Have you tried registering your Cassandra tables in Hive/Spark SQL using the data frames API. These should be then available to query via the Spark SQL/Thrift JDBC Server. Deenar On 1 June 2015 at 19:33, Mohammed Guller wrote: > Nobody using Spark SQL JDBC/Thrift server with

RE: Anybody using Spark SQL JDBC server with DSE Cassandra?

Nobody using Spark SQL JDBC/Thrift server with DSE Cassandra? Mohammed From: Mohammed Guller [mailto:moham...@glassbeam.com] Sent: Friday, May 29, 2015 11:49 AM To: user@spark.apache.org Subject: Anybody using Spark SQL JDBC server with DSE Cassandra? Hi - We have successfully integrated Spark

Anybody using Spark SQL JDBC server with DSE Cassandra?

Spark SQL JDBC server. However, I have been unable to find a driver that would allow the Spark SQL Thrift/JDBC server to connect with Cassandra. DataStax provides a closed-source driver that comes only with the DSE version of Cassandra. I would like to find out how many people are using the Spark

Re: What's the advantage features of Spark SQL(JDBC)

To: Cheng, Hao; User Subject: Re: What's the advantage features of Spark SQL(JDBC)   @Hao, As you said, there is no advantage feature for JDBC, it just provides unified api to support different data sources. Is it right?     On Friday, May 15, 2015 2:46 PM, "Cheng, Hao" wrot

RE: What's the advantage features of Spark SQL(JDBC)

Yes. From: Yi Zhang [mailto:zhangy...@yahoo.com] Sent: Friday, May 15, 2015 2:51 PM To: Cheng, Hao; User Subject: Re: What's the advantage features of Spark SQL(JDBC) @Hao, As you said, there is no advantage feature for JDBC, it just provides unified api to support different data sources.

Re: What's the advantage features of Spark SQL(JDBC)

er Subject: What's the advantage features of Spark SQL(JDBC)   Hi All,   Comparing direct access via JDBC, what's the advantage features of Spark SQL(JDBC) to access external data source?   Any tips are welcome! Thanks.   Regards, Yi  

RE: What's the advantage features of Spark SQL(JDBC)

Spark SQL just take the JDBC as a new data source, the same as we need to support loading data from a .csv or .json. From: Yi Zhang [mailto:zhangy...@yahoo.com.INVALID] Sent: Friday, May 15, 2015 2:30 PM To: User Subject: What's the advantage features of Spark SQL(JDBC) Hi All, Comp

What's the advantage features of Spark SQL(JDBC)

Hi All, Comparing direct access via JDBC, what's the advantage features of Spark SQL(JDBC) to access external data source? Any tips are welcome! Thanks. Regards,Yi

Re: Spark-SQL JDBC driver

rk SQL (i.e. Spark Shell or Spark-SQL.sh) is not visible on thrift, >>> since thrift has its own session containing its own RDD. >>> >>> Spark SQL experts on the forum can confirm on this though. >>> >>> >>> >>> *From:* Cheng Lian [mailt

Re: Spark-SQL JDBC driver

> >> >> >> *From:* Cheng Lian [mailto:lian.cs@gmail.com] >> *Sent:* Tuesday, December 9, 2014 6:42 AM >> *To:* Anas Mosaad >> *Cc:* Judy Nash; user@spark.apache.org >> *Subject:* Re: Spark-SQL JDBC driver >> >> >> >> Acc

RE: Spark-SQL JDBC driver

> > Spark SQL experts on the forum can confirm on this though. > > > > *From:* Cheng Lian [mailto:lian.cs@gmail.com] > *Sent:* Tuesday, December 9, 2014 6:42 AM > *To:* Anas Mosaad > *Cc:* Judy Nash; user@spark.apache.org > *Subject:* Re: Spark-SQL JDBC driver > &

RE: Spark-SQL JDBC driver

SQL experts on the forum can confirm on this though. From: Cheng Lian [mailto:lian.cs@gmail.com] Sent: Tuesday, December 9, 2014 6:42 AM To: Anas Mosaad Cc: Judy Nash; user@spark.apache.org Subject: Re: Spark-SQL JDBC driver According to the stacktrace, you were still using SQLContext rather

Re: Spark-SQL JDBC driver

park-shell (i.e. like any other relational database) On Tue, Dec 9, 2014 at 11:05 AM, Cheng Lian mailto:lian.cs@gmail.com>> wrote: Essentially, the Spark SQL JDBC Thrift server is just a Spark port of HiveServer2. You don't need to run Hive, but you

Re: Spark-SQL JDBC driver

-+* > > *+-+* > > *No rows selected (0.106 seconds)* > > *0: jdbc:hive2://localhost:1> * > > > > Kindly advice, what am I missing? I want to read the RDD using SQL from > outside spark-shell (i.e. like any other relational database) >

Re: Spark-SQL JDBC driver

>> wrote: Essentially, the Spark SQL JDBC Thrift server is just a Spark port of HiveServer2. You don't need to run Hive, but you do need a working Metastore. On 12/9/14 3:59 PM, Anas Mosaad wrote: Thanks Judy, this is exactly what I'm looking for. However, and

Re: Spark-SQL JDBC driver

missing? I want to read the RDD using SQL from outside spark-shell (i.e. like any other relational database) On Tue, Dec 9, 2014 at 11:05 AM, Cheng Lian wrote: > Essentially, the Spark SQL JDBC Thrift server is just a Spark port of > HiveServer2. You don't need to run Hive, but you

Re: Spark-SQL JDBC driver

Essentially, the Spark SQL JDBC Thrift server is just a Spark port of HiveServer2. You don't need to run Hive, but you do need a working Metastore. On 12/9/14 3:59 PM, Anas Mosaad wrote: Thanks Judy, this is exactly what I'm looking for. However, and plz forgive me if it's a d

Re: Spark-SQL JDBC driver

rta.com] > *Sent:* Monday, December 8, 2014 11:01 AM > *To:* user@spark.apache.org > *Subject:* Spark-SQL JDBC driver > > > > Hello Everyone, > > > > I'm brand new to spark and was wondering if there's a JDBC driver to > access spark-SQL directly. I'm

RE: Spark-SQL JDBC driver

Subject: Spark-SQL JDBC driver Hello Everyone, I'm brand new to spark and was wondering if there's a JDBC driver to access spark-SQL directly. I'm running spark in standalone mode and don't have hadoop in this environment. -- Best Regards/أطيب المنى, Anas Mosaad

Spark-SQL JDBC driver

Hello Everyone, I'm brand new to spark and was wondering if there's a JDBC driver to access spark-SQL directly. I'm running spark in standalone mode and don't have hadoop in this environment. -- *Best Regards/أطيب المنى,* *Anas Mosaad*

RE: Spark SQL JDBC

...@spark.incubator.apache.org Subject: Re: Spark SQL JDBC When you re-ran sbt did you clear out the packages first and ensure that the datanucleus jars were generated within lib_managed? I remembered having to do that when I was working testing out different configs. On Thu, Sep 11, 2014 at 10:50 AM

Re: Spark SQL JDBC

wrote: > Even when I comment out those 3 lines, I still get the same error. Did > someone solve this? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-JDBC-tp11369p13992.html > Sent from the Apache Spark

Re: Spark SQL JDBC

Even when I comment out those 3 lines, I still get the same error. Did someone solve this? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-JDBC-tp11369p13992.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: Spark SQL JDBC

Oh, thanks for reporting this. This should be a bug since SPARK_HIVE was deprecated, we shouldn’t rely on it any more. ​ On Wed, Aug 13, 2014 at 1:23 PM, ZHENG, Xu-dong wrote: > Just find this is because below lines in make_distribution.sh doesn't work: > > if [ "$SPARK_HIVE" == "true" ]; then

Re: Spark SQL JDBC

Just find this is because below lines in make_distribution.sh doesn't work: if [ "$SPARK_HIVE" == "true" ]; then cp "$FWDIR"/lib_managed/jars/datanucleus*.jar "$DISTDIR/lib/" fi There is no definition of $SPARK_HIVE in make_distribution.sh. I should set it explicitly. On Wed, Aug 13, 2014 at

Re: Spark SQL JDBC

Hi Cheng, I also meet some issues when I try to start ThriftServer based a build from master branch (I could successfully run it from the branch-1.0-jdbc branch). Below is my build command: ./make-distribution.sh --skip-java-test -Phadoop-2.4 -Phive -Pyarn -Dyarn.version=2.4.0 -Dhadoop.version=2.

Re: Spark SQL JDBC

Hive pulls in a ton of dependencies that we were afraid would break existing spark applications. For this reason all hive submodules are optional. On Tue, Aug 12, 2014 at 7:43 AM, John Omernik wrote: > Yin helped me with that, and I appreciate the onlist followup. A few > questions: Why is th

Re: Spark SQL JDBC

Yin helped me with that, and I appreciate the onlist followup. A few questions: Why is this the case? I guess, does building it with thriftserver add much more time/size to the final build? It seems that unless documented well, people will miss that and this situation would occur, why would we no

Re: Spark SQL JDBC

Hi John, the JDBC Thrift server resides in its own build profile and need to be enabled explicitly by ./sbt/sbt -Phive-thriftserver assembly. ​ On Tue, Aug 5, 2014 at 4:54 AM, John Omernik wrote: > I am using spark-1.1.0-SNAPSHOT right now and trying to get familiar with > the JDBC thrift serve

Spark SQL JDBC

I am using spark-1.1.0-SNAPSHOT right now and trying to get familiar with the JDBC thrift server. I have everything compiled correctly, I can access data in spark-shell on yarn from my hive installation. Cached tables, etc all work. When I execute ./sbin/start-thriftserver.sh I get the error bel

Re: Spark SQL JDBC Connectivity

://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-JDBC-Connectivity-tp6511p10986.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >

Re: Spark SQL JDBC Connectivity

now. It is easy to do this and took a just a few hours and it works for our use case. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-JDBC-Connectivity-tp6511p10986.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Spark SQL JDBC Connectivity and more

> [Venkat] Are you saying - pull in the SharkServer2 code in my standalone > spark application (as a part of the standalone application process), pass > in > the spark context of the standalone app to SharkServer2 Sparkcontext at > startup and viola we get a SQL/JDBC interfaces for the RDDs of t

Re: Spark SQL JDBC Connectivity and more

s tables? Thanks for the clarification. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-JDBC-Connectivity-tp6511p7264.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Spark SQL JDBC Connectivity and more

On Thu, May 29, 2014 at 3:26 PM, Venkat Subramanian wrote: > > 1) If I have a standalone spark application that has already built a RDD, > how can SharkServer2 or for that matter Shark access 'that' RDD and do > queries on it. All the examples I have seen for Shark, the RDD (tables) are > created

Re: Spark SQL JDBC Connectivity and more

may be a very common use case? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-JDBC-Connectivity-tp6511p6543.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Spark SQL JDBC Connectivity

On Wed, May 28, 2014 at 11:39 PM, Venkat Subramanian wrote: > We are planning to use the latest Spark SQL on RDDs. If a third party > application wants to connect to Spark via JDBC, does Spark SQL have > support? > (We want to avoid going though Shark/Hive JDBC layer as we need good > performance)

Spark SQL JDBC Connectivity

.nabble.com/Spark-SQL-JDBC-Connectivity-tp6511.html Sent from the Apache Spark User List mailing list archive at Nabble.com.