Thanks Wenchen! On Wed, Dec 18, 2019 at 7:25 PM Wenchen Fan <cloud0...@gmail.com> wrote:
> Hi Aakash, > > You can try the latest DS v2 with the 3.0 preview, and the API is in a > quite stable shape now. With the latest API, a Writer is created from a > Table, and the Table has the partitioning information. > > Thanks, > Wenchen > > On Wed, Dec 18, 2019 at 3:22 AM aakash aakash <email2aak...@gmail.com> > wrote: > >> Thanks Andrew! >> >> It seems there is a drastic change in 3.0, going through it. >> >> -Aakash >> >> On Tue, Dec 17, 2019 at 11:01 AM Andrew Melo <andrew.m...@gmail.com> >> wrote: >> >>> Hi Aakash >>> >>> On Tue, Dec 17, 2019 at 12:42 PM aakash aakash <email2aak...@gmail.com> >>> wrote: >>> >>>> Hi Spark dev folks, >>>> >>>> First of all kudos on this new Data Source v2, API looks simple and it >>>> makes easy to develop a new data source and use it. >>>> >>>> With my current work, I am trying to implement a new data source V2 >>>> writer with Spark 2.3 and I was wondering how I will get the info about >>>> partition by columns. I see that it has been passed to Data Source V1 from >>>> DataFrameWriter but not for V2. >>>> >>> >>> Not directly related to your Q, but just so you're aware, the DSv2 API >>> evolved from 2.3->2.4 and then again for 2.4->3.0. >>> >>> Cheers >>> Andrew >>> >>> >>>> >>>> >>>> Thanks, >>>> Aakash >>>> >>>
