I see, thanks for the info!
On Mon, Jan 23, 2017 at 4:12 PM, Xiao Li wrote:
> Reynold mentioned the direction we are heading. You can see many PRs the
> community submitted are for this target. To achieve this, a lot of works we
> need to do.
>
> For example, for some serde, Hive metastore will
Reynold mentioned the direction we are heading. You can see many PRs the
community submitted are for this target. To achieve this, a lot of works we
need to do.
For example, for some serde, Hive metastore will infer the schema when the
schema is not provided, but our InMemoryCatalog does not have
Cool, thanks for the info.
I think this is something we are going to change to completely decouple the
> Hive support and catalog.
Is there a ticket for this? I did a search in jira and only found
"SPARK-16275: Implement all the Hive fallback functions", which seems to be
related to it.
On Mon
Agree. : )
2017-01-22 11:20 GMT-08:00 Reynold Xin :
> To be clear there are two separate "hive" we are talking about here. One
> is the catalog, and the other is the Hive serde and UDF support. We want to
> get to a point that the choice of catalog does not impact the functionality
> in Spark oth
To be clear there are two separate "hive" we are talking about here. One is
the catalog, and the other is the Hive serde and UDF support. We want to
get to a point that the choice of catalog does not impact the functionality
in Spark other than where the catalog is stored.
On Sun, Jan 22, 2017 at
We have a pending PR to block users to create the Hive serde table when
using InMemroyCatalog. See: https://github.com/apache/spark/pull/16587 I
believe it answers your question.
BTW, we still can create the regular data source tables and insert the data
into the tables. The major difference is wh
I think this is something we are going to change to completely decouple the
Hive support and catalog.
On Sun, Jan 22, 2017 at 4:51 AM Shuai Lin wrote:
> Hi all,
>
> Currently when the in-memory catalog is used, e.g. through `--conf
> spark.sql.catalogImplementation=in-memory`, we can create a p
