[Discuss] Datasource v2 support for manipulating partitions

I've been following the development of the new data source abstraction with keen interest. One of the issues that has occurred to me as I sat down and planned how I would implement a data source is how I would support manipulating partitions. My reading of the current prototype is that Data sourc

[Discuss] Datasource v2 support for Kerberos

The current V2 Datasource API provides support for querying a portion of the SparkConfig namespace (spark.datasource.*) via the SessionConfigSupport API. This was designed with the assumption that all configuration information for v2 data sources should be separate from each other. Unfortunately,

Re: [Discuss] Datasource v2 support for manipulating partitions

Hi Jayesh, I get where you are coming from - partitions are just an implementation optimisation that we really shouldn’t be bothering the end user with. Unfortunately that view is like saying RPC is like a procedure call, and details of the network transport should be hidden from the end user. COR

Re: [Discuss] Datasource v2 support for Kerberos

I believe the current spark config system is unfortunate in the way it has grown - you have no way of telling which sub-systems uses which configuration options without direct and detailed reading of the code. Isolating config items for datasources into a separate namespaces (rather then using a w

Re: [Discuss] Datasource v2 support for Kerberos

I like the shared namespace option better then the white listing option for any newly defined configuration information. All of the Kerberos options already exist in their own legacy locations though - changing their location could break a lot of systems. Perhaps we can use the shared namespace

Re: [Discuss] Datasource v2 support for Kerberos

To give some Kerberos specific examples, The spark-submit args: -–conf spark.yarn.keytab=path_to_keytab -–conf spark.yarn.principal=princi...@realm.com are currently not passed through to the data sources. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ -

Re: [Discuss] Datasource v2 support for Kerberos

Hi Steve, I think that passing a kerberos keytab around is one of those bad ideas that is entirely appropriate to re-question every single time you come across it. It has been used already in spark when interacting with Kerberos systems that do not support delegation tokens. Any such system will e