> Is this something that would be exposed/relevant to the Python API? Or is
this just for people implementing their own Spark data source?
It's latter, and it also helps simplifying built-in data sources as well
(as I found the needs while working on
https://github.com/apache/spark/pull/26845)
On
Nice, thanks for the answer! I'll craft a PR soon. Thanks again.
On Thu, Dec 12, 2019 at 3:32 AM Ryan Blue wrote:
> Sounds good to me, too.
>
> On Wed, Dec 11, 2019 at 1:18 AM Jungtaek Lim
> wrote:
>
>> Thanks for the quick response, Wenchen!
>>
>> I'll leave this thread for early tomorrow so t
Is this something that would be exposed/relevant to the Python API? Or is
this just for people implementing their own Spark data source?
On Wed, Dec 11, 2019 at 12:35 AM Jungtaek Lim
wrote:
> Hi devs,
>
> I'd like to propose to add close() on DataWriter explicitly, which is the
> place for resou
Sounds good to me, too.
On Wed, Dec 11, 2019 at 1:18 AM Jungtaek Lim
wrote:
> Thanks for the quick response, Wenchen!
>
> I'll leave this thread for early tomorrow so that someone in US timezone
> can chime in, and craft a patch if no one objects.
>
> On Wed, Dec 11, 2019 at 4:41 PM Wenchen Fan
Thanks for the quick response, Wenchen!
I'll leave this thread for early tomorrow so that someone in US timezone
can chime in, and craft a patch if no one objects.
On Wed, Dec 11, 2019 at 4:41 PM Wenchen Fan wrote:
> PartitionReader extends Closable, seems reasonable to me to do the same
> for
PartitionReader extends Closable, seems reasonable to me to do the same
for DataWriter.
On Wed, Dec 11, 2019 at 1:35 PM Jungtaek Lim
wrote:
> Hi devs,
>
> I'd like to propose to add close() on DataWriter explicitly, which is the
> place for resource cleanup.
>
> The rationalization of the propos
Hi devs,
I'd like to propose to add close() on DataWriter explicitly, which is the
place for resource cleanup.
The rationalization of the proposal is due to the lifecycle of DataWriter.
If the scaladoc of DataWriter is correct, the lifecycle of DataWriter
instance ends at either commit() or abort
