Re: PySpark: Make persist() return a context manager

i think it limits the usability of with statement. and it could be somewhat confusing because of this, so i would mention it in docs. i like the idea though. On Fri, Aug 5, 2016 at 7:04 PM, Nicholas Chammas wrote: > Good point. > > Do you think it's sufficient to note this somewhere in the docu

Re: PySpark: Make persist() return a context manager

Good point. Do you think it's sufficient to note this somewhere in the documentation (or simply assume that user understanding of transformations vs. actions means they know this), or are there other implications that need to be considered? On Fri, Aug 5, 2016 at 6:50 PM Koert Kuipers wrote: >

Re: PySpark: Make persist() return a context manager

The tricky part is that the action needs to be inside the with block, not just the transformation that uses the persisted data. On Aug 5, 2016 1:44 PM, "Nicholas Chammas" wrote: Okie doke, I've filed a JIRA for this here: https://issues.apache. org/jira/browse/SPARK-16921 On Fri, Aug 5, 2016 at

Re: PySpark: Make persist() return a context manager

Okie doke, I've filed a JIRA for this here: https://issues.apache.org/jira/browse/SPARK-16921 On Fri, Aug 5, 2016 at 2:08 AM Reynold Xin wrote: > Sounds like a great idea! > > On Friday, August 5, 2016, Nicholas Chammas > wrote: > >> Context managers >>

Re: PySpark: Make persist() return a context manager

Sounds like a great idea! On Friday, August 5, 2016, Nicholas Chammas wrote: > Context managers > are > a natural way to capture closely related setup and teardown code in Python. > > For example, they are commonly used when d

PySpark: Make persist() return a context manager

Context managers are a natural way to capture closely related setup and teardown code in Python. For example, they are commonly used when doing file I/O: with open('/path/to/file') as f: contents = f.read() ... Once th