Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-31 Thread James Baker
;d consider that out of scope for the current one. Otherwise we'd not release something probably for the next 2 or 3 years. On Wed, Aug 30, 2017 at 11:50 PM, James Baker mailto:j.ba...@outlook.com>> wrote: I guess I was more suggesting that by coding up the powerful mode as the API,

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-31 Thread James Baker
;d consider that out of scope for the current one. Otherwise we'd not release something probably for the next 2 or 3 years. On Wed, Aug 30, 2017 at 11:50 PM, James Baker mailto:j.ba...@outlook.com>> wrote: I guess I was more suggesting that by coding up the powerful mode as the API,

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-31 Thread James Baker
;d consider that out of scope for the current one. Otherwise we'd not release something probably for the next 2 or 3 years. On Wed, Aug 30, 2017 at 11:50 PM, James Baker mailto:j.ba...@outlook.com>> wrote: I guess I was more suggesting that by coding up the powerful mode as the API,

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-31 Thread James Baker
;d consider that out of scope for the current one. Otherwise we'd not release something probably for the next 2 or 3 years. On Wed, Aug 30, 2017 at 11:50 PM, James Baker mailto:j.ba...@outlook.com>> wrote: I guess I was more suggesting that by coding up the powerful mode as the API,

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-30 Thread James Baker
ble, to save the effort of designing a stable representation of query plan and maintaining forward compatibility. On Wed, Aug 30, 2017 at 10:53 AM, James Baker mailto:j.ba...@outlook.com>> wrote: I'll just focus on the one-by-one thing for now - it's the thing that blocks me the m

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-30 Thread James Baker
ble, to save the effort of designing a stable representation of query plan and maintaining forward compatibility. On Wed, Aug 30, 2017 at 10:53 AM, James Baker mailto:j.ba...@outlook.com>> wrote: I'll just focus on the one-by-one thing for now - it's the thing that blocks me the m

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-30 Thread James Baker
ble, to save the effort of designing a stable representation of query plan and maintaining forward compatibility. On Wed, Aug 30, 2017 at 10:53 AM, James Baker mailto:j.ba...@outlook.com>> wrote: I'll just focus on the one-by-one thing for now - it's the thing that blocks me the m

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-30 Thread James Baker
ble, to save the effort of designing a stable representation of query plan and maintaining forward compatibility. On Wed, Aug 30, 2017 at 10:53 AM, James Baker mailto:j.ba...@outlook.com>> wrote: I'll just focus on the one-by-one thing for now - it's the thing that blocks me the m

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-29 Thread James Baker
port interface, so that advanced data source can implement it for optimal performance, and simple data source doesn't need to care about it and keep simple. The current design is very friendly to simple data source, and has the potential to support complex data source, I prefer the current

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-29 Thread James Baker
port interface, so that advanced data source can implement it for optimal performance, and simple data source doesn't need to care about it and keep simple. The current design is very friendly to simple data source, and has the potential to support complex data source, I prefer the current

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-29 Thread James Baker
port interface, so that advanced data source can implement it for optimal performance, and simple data source doesn't need to care about it and keep simple. The current design is very friendly to simple data source, and has the potential to support complex data source, I prefer the current

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-29 Thread James Baker
posed to be. If a data source simply implements one of the trait, it'd be logically identical to v1. I don't see why it would be worse or better, other than v2 provides much stronger forward compatibility guarantees than v1. On Tue, Aug 29, 2017 at 4:54 AM, James Baker mailto:j.ba..

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-29 Thread James Baker
posed to be. If a data source simply implements one of the trait, it'd be logically identical to v1. I don't see why it would be worse or better, other than v2 provides much stronger forward compatibility guarantees than v1. On Tue, Aug 29, 2017 at 4:54 AM, James Baker mailto:j.ba..

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-28 Thread James Baker
Copying from the code review comments I just submitted on the draft API (https://github.com/cloud-fan/spark/pull/10#pullrequestreview-59088745): Context here is that I've spent some time implementing a Spark datasource and have had some issues with the current API which are made worse in V2. Th

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-28 Thread James Baker
Copying from the code review comments I just submitted on the draft API (https://github.com/cloud-fan/spark/pull/10#pullrequestreview-59088745): Context here is that I've spent some time implementing a Spark datasource and have had some issues with the current API which are made worse in V2. Th

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

2017-08-28 Thread James Baker
Copying from the code review comments I just submitted on the draft API (https://github.com/cloud-fan/spark/pull/10#pullrequestreview-59088745): Context here is that I've spent some time implementing a Spark datasource and have had some issues with the current API which are made worse in V2. Th