Re: DataSourceV2 sync notes - 2 October 2019

Hi Ryan, Thanks for summarizing and sending out the notes! I've created the JIRA ticket to add v2 statements for all the commands that need to resolve a table: https://issues.apache.org/jira/browse/SPARK-29481 Contributions to it are appreciated! Thanks, Wenchen On Fri, Oct 11, 2019 at 7:05 AM

DataSourceV2 sync notes - 2 October 2019

Here are my notes from last week's DSv2 sync. *Attendees*: Ryan Blue Terry Kim Wenchen Fan *Topics*: - SchemaPruning only supports Parquet and ORC? - Out of order optimizer rules - 3.0 work - Rename session catalog to spark_catalog - Finish TableProvider update to avoid ano

DataSourceV2 sync notes - 24 July 2019

Here are my notes from the last DSv2 sync. Sorry it's a bit late! *Attendees*: Ryan Blue John Zhuge Raynmond McCollum Terry Kim Gengliang Wang Jose Torres Wenchen Fan Priyanka Gomatam Matt Cheah Russel Spitzer Burak Yavuz *Topics*: - Check in on blockers - Remove SaveMode - Reorg

Re: DataSourceV2 sync notes - 10 July 2019

I agree that the long-term solution is much farther away, but I'm not sure it is a good idea to do this in the optimizer. Maybe we could find a good way to do it, but the initial complication required before we moved to push-down to the conversion to physical plan was really bad. Plus, this has bee

Re: DataSourceV2 sync notes - 10 July 2019

Hi Ryan, Thanks for summarizing and sending out the meeting notes! Unfortunately, I missed the last sync, but the topics are really interesting, especially the stats integration. The ideal solution I can think of is to refactor the optimizer/planner and move all the stats-based optimization to th

DataSourceV2 sync notes - 10 July 2019

Here are my notes from the last sync. If you’d like to be added to the invite or have topics, please let me know. *Attendees*: Ryan Blue Matt Cheah Yifei Huang Jose Torres Burak Yavuz Gengliang Wang Michael Artz Russel Spitzer *Topics*: - Existing PRs - V2 session catalog: https://gith

DataSourceV2 sync notes - 12 June 2019

Here are the latest DSv2 sync notes. Please reply with updates or corrections. *Attendees*: Ryan Blue Michael Armbrust Gengliang Wang Matt Cheah John Zhuge *Topics*: Wenchen’s reorganization proposal Problems with TableProvider - property map isn’t sufficient New PRs: - ReplaceTable: https

DataSourceV2 sync notes - 29 May 2019

Here are my notes from last night’s sync. I had to leave early, so there may be more discussion. Others can fill in the details for those topics. *Attendees*: John Zhuge Ryan Blue Yifei Huang Matt Cheah Yuanjian Li Russell Spitzer Kevin Yu *Topics*: - Atomic extensions for the TableCatalog A

DataSourceV2 sync notes - 15 May 2019

Sorry these notes are so late, I didn’t get to the write up until now. As usual, if anyone has corrections or comments, please reply. *Attendees*: John Zhuge Ryan Blue Andrew Long Wenchen Fan Gengliang Wang Russell Spitzer Yuanjian Li Yifei Huang Matt Cheah Amardeep Singh Dhilon Zhilmil Dhion Rya

Re: DataSourceV2 sync notes - 20 Feb 2019

Thanks Ryan! On Tue, Mar 5, 2019 at 7:19 PM Ryan Blue wrote: > Everyone is welcome to join this discussion. Just send me an e-mail to get > added to the invite. > > Stavros, I'll add you. > > rb > > On Tue, Mar 5, 2019 at 5:43 AM Stavros Kontopoulos < > stavros.kontopou...@lightbend.com> wrote:

Re: DataSourceV2 sync notes - 20 Feb 2019

Everyone is welcome to join this discussion. Just send me an e-mail to get added to the invite. Stavros, I'll add you. rb On Tue, Mar 5, 2019 at 5:43 AM Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: > Thanks for the update, is this meeting open for other people to join? > > St

Re: DataSourceV2 sync notes - 20 Feb 2019

Thanks for the update, is this meeting open for other people to join? Stavros On Thu, Feb 21, 2019 at 10:56 PM Ryan Blue wrote: > Here are my notes from the DSv2 sync last night. As always, if you have > corrections, please reply with them. And if you’d like to be included on > the invite to pa

DataSourceV2 sync notes - 20 Feb 2019

Here are my notes from the DSv2 sync last night. As always, if you have corrections, please reply with them. And if you’d like to be included on the invite to participate in the next sync (6 March), send me an email. Here’s a quick summary of the topics where we had consensus last night: - The

DataSourceV2 sync notes

Hi everyone, here are notes from the last DSv2 sync on 23 January 2019. Here are the highlights: - Agreed that using v2 should not change behavior for file sources. (Cannot make this guarantee for all v1 sources) - Consensus for the approach proposed on the dev list for identifying ta

DataSourceV2 sync notes

Here are my notes from the DSv2 sync last night. *As usual, I didn’t take great notes because I was participating in the discussion. Feel free to send corrections or clarification.* *Attendees*: Ryan Blue John Zhuge Xiao Li Reynold Xin Felix Cheung Anton Okolnychyi Bruce Robbins Dale Richardson D

Re: DataSourceV2 sync notes (#4)

Thanks for sending out the meeting notes from last week's discussion Ryan! For technical unknown reasons, I could not unmute myself and be heard when I was trying to pitch in during one of the topic discussions regarding default value handling for traditional databases. Had posted response in ch

DataSourceV2 sync notes (#4)

Hi everyone, sorry these notes are late. I didn’t have the time to write this up last week. For anyone interested in the next sync, we decided to skip next week and resume in early January. I’ve already sent the invite. As usual, if you have topics you’d like to discuss or would like to be added t