Re: A new external catalog

2018-02-16 Thread Steve Loughran
On 14 Feb 2018, at 19:56, Tayyebi, Ameen mailto:tayye...@amazon.com>> wrote: Newbie question: I want to add system/integration tests for the new functionality. There are a set of existing tests around Spark Catalog that I can leverage. Great. The provider I’m writing is backed by a web servi

Re: A new external catalog

2018-02-16 Thread Steve Loughran
On 14 Feb 2018, at 13:51, Tayyebi, Ameen mailto:tayye...@amazon.com>> wrote: Thanks a lot Steve. I’ll go through the Jira’s you linked in detail. I took a quick look and am sufficiently scared for now. I had run into that warning from the S3 stream before. Sigh. things like that are trouble

Re: A new external catalog

2018-02-14 Thread Tayyebi, Ameen
, -Ameen From: Steve Loughran Date: Tuesday, February 13, 2018 at 5:01 PM To: "Tayyebi, Ameen" Cc: Apache Spark Dev Subject: Re: A new external catalog On 13 Feb 2018, at 21:20, Tayyebi, Ameen mailto:tayye...@amazon.com>> wrote: Yes, I’m thinking about upgrading to these: 1.9.0

Re: A new external catalog

2018-02-14 Thread Tayyebi, Ameen
Spark Dev Subject: Re: A new external catalog On 13 Feb 2018, at 21:20, Tayyebi, Ameen mailto:tayye...@amazon.com>> wrote: Yes, I’m thinking about upgrading to these: 1.9.0 1.11.272 From: 1.7.3 1.11.76 272 is the earliest that has Glue. How about I let the build system run the tes

Re: A new external catalog

2018-02-13 Thread Steve Loughran
;Tayyebi, Ameen" mailto:tayye...@amazon.com>> Cc: Apache Spark Dev mailto:dev@spark.apache.org>> Subject: Re: A new external catalog On 13 Feb 2018, at 19:50, Tayyebi, Ameen mailto:tayye...@amazon.com>> wrote: The biggest challenge is that I had to upgrade the AWS SDK to a

Re: A new external catalog

2018-02-13 Thread Tayyebi, Ameen
at 3:34 PM To: "Tayyebi, Ameen" Cc: Apache Spark Dev Subject: Re: A new external catalog On 13 Feb 2018, at 19:50, Tayyebi, Ameen mailto:tayye...@amazon.com>> wrote: The biggest challenge is that I had to upgrade the AWS SDK to a newer version so that it includes the Glue c

Re: A new external catalog

2018-02-13 Thread Steve Loughran
On 13 Feb 2018, at 19:50, Tayyebi, Ameen mailto:tayye...@amazon.com>> wrote: The biggest challenge is that I had to upgrade the AWS SDK to a newer version so that it includes the Glue client since Glue is a new service. So far, I haven’t see any jar hell issues, but that’s the main drawback

A new external catalog

2018-02-13 Thread Tayyebi, Ameen
Hello everyone, For those of you not familiar with AWS Glue Catalog, it’s a Hive Metastore implemented as a web service. The Glue service is composed of different components, but the one I’m interested in is the Catalog. Today, there’s a Hive metastore implementat