> Is this something that would be exposed/relevant to the Python API? Or is
this just for people implementing their own Spark data source?
It's latter, and it also helps simplifying built-in data sources as well
(as I found the needs while working on
https://github.com/apache/spark/pull/26845)
On
Nice, thanks for the answer! I'll craft a PR soon. Thanks again.
On Thu, Dec 12, 2019 at 3:32 AM Ryan Blue wrote:
> Sounds good to me, too.
>
> On Wed, Dec 11, 2019 at 1:18 AM Jungtaek Lim
> wrote:
>
>> Thanks for the quick response, Wenchen!
>>
>> I'll leave this thread for early tomorrow so t
Is this something that would be exposed/relevant to the Python API? Or is
this just for people implementing their own Spark data source?
On Wed, Dec 11, 2019 at 12:35 AM Jungtaek Lim
wrote:
> Hi devs,
>
> I'd like to propose to add close() on DataWriter explicitly, which is the
> place for resou
Sounds good to me, too.
On Wed, Dec 11, 2019 at 1:18 AM Jungtaek Lim
wrote:
> Thanks for the quick response, Wenchen!
>
> I'll leave this thread for early tomorrow so that someone in US timezone
> can chime in, and craft a patch if no one objects.
>
> On Wed, Dec 11, 2019 at 4:41 PM Wenchen Fan
Thanks for the quick response, Wenchen!
I'll leave this thread for early tomorrow so that someone in US timezone
can chime in, and craft a patch if no one objects.
On Wed, Dec 11, 2019 at 4:41 PM Wenchen Fan wrote:
> PartitionReader extends Closable, seems reasonable to me to do the same
> for
PartitionReader extends Closable, seems reasonable to me to do the same
for DataWriter.
On Wed, Dec 11, 2019 at 1:35 PM Jungtaek Lim
wrote:
> Hi devs,
>
> I'd like to propose to add close() on DataWriter explicitly, which is the
> place for resource cleanup.
>
> The rationalization of the propos
