[issue44604] [Enhancement] Asyncio task decorator to provide interface to define async DAGs (similar to dask's delayed interface)

[Art]

Change by Art :


--
title: [Enhancement] Asyncio task decorator to provide functionality similar to 
dask's delayed interface -> [Enhancement] Asyncio task decorator to provide 
interface to define async DAGs (similar to dask's delayed interface)

___
Python tracker 
<https://bugs.python.org/issue44604>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44604] [Enhancement] Asyncio task decorator to provide interface to define async DAGs (similar to dask's delayed interface)

[Art]

Art  added the comment:

I understand the hesitation to add this to the library, however I'd like to 
make a case for why I believe that it fits. To start this is my understanding 
of the scope of asyncio (or at least part of the scope, taken from the asyncio 
docs):

asyncio provides a set of high-level APIs to:
- run Python coroutines concurrently and have full control over their execution;
- perform network IO and IPC;
- control subprocesses;
- distribute tasks via queues;
- synchronize concurrent code;

I believe this decorator would be another high-level api that would be 
considered a valid, useful, simple, and if it was more well-known, popular, 
method of exerting "full control over their execution".

Here's an example of a few places from around the internet where people are 
asking how to do something that would directly benefit from this API, and where 
I believe frankly that this API would provide a much cleaner/better solution 
than those proposed.

https://old.reddit.com/r/learnpython/comments/oi3ueo/how_do_you_use_asyncio_to_do_this/
(inspiration for demo solution provided above)
https://stackoverflow.com/questions/58581270/what-is-the-cleanest-way-to-write-and-run-a-dag-of-tasks
(user specifies their tasks are IO bound and that they tried to use 
asyncio, but to no avail, or it was too complicated)

In both these cases above, it is explicitly implied that the user tried to use 
asyncio to solve their problem, but could not find a satisfactory way to do so.

Here's a few more examples of asyncio code, that aren't as directly related to 
the issue at hand, but would greatly benefit from this isolated "task" 
structure.

https://stackoverflow.com/questions/58905515/how-to-ensure-python-s-asyncio-library-executes-tasks-in-depth-first-search-orde
(user trying to build/execute a DAG of async tasks)
https://stackoverflow.com/questions/67481509/how-should-i-design-a-concurrency-pipeline-with-asyncio
(user could/should consider their task pipeline as a DAG which would 
conceptually simplify what they are trying to achieve)

To directly address the concern of "why add this to asyncio, and not a 
third-party library".
1) I believe this falls into the scope of the library as demonstrated in the 
first paragraph of this comment
2) Users look to and trust asyncio to provide good high level apis to build 
better code/systems (As you can see from some of the examples I posted above)
3) This is a very simple high-level api, that will not interfere with anything 
else in the library. Will introduce no compatibility issues
4) This point is conjecture, but I believe that this api will keep users from 
unnecessarily moving their workflows to other libraries like dask (which I love 
and contribute too) or airflow as you can see from some of the examples where 
users claim to try other libraries for something that asyncio could easily 
handle.
5) Inclusion in asyncio (over a third-party library) would provide faster, more 
reputable exposure to users of a great method of thinking about/solving their 
problems.

I hope you'll re-consider your position, but completely understand if not, in 
which case I'll close this issue.

--

___
Python tracker 
<https://bugs.python.org/issue44604>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44604] [Enhancement] Asyncio task decorator to provide interface to define async DAGs (similar to dask's delayed interface)

[Art]

Art  added the comment:

Note: regarding my inclusion of "airflow" in point #4 above. I in no way 
believe that airflow is a viable alternative to this proposal. I included it 
because one of those posts I linked explicitly mentioned that they looked into 
it as an alternative.

--

___
Python tracker 
<https://bugs.python.org/issue44604>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44604] [Enhancement] Asyncio task decorator to provide interface to define async DAGs (similar to dask's delayed interface)

[Art]

Art  added the comment:

Hi Yury, could you by any chance point me to some material on the inclusion of 
task groups in 3.11? I've found these "task group" docs from a library called 
AnyIO: https://anyio.readthedocs.io/en/stable/tasks.html.

If the asyncio task group feature is similar to this, then while I think it is 
a nice inclusion, I believe it does not cover the use case that this proposal 
would cover.

* If your opinion on the matter is still not final, then would you be open to a 
further demonstration of this proposal? I could gather several use cases from 
SO and other sites and demo how they could be elegantly solved with this 
interface.

--

___
Python tracker 
<https://bugs.python.org/issue44604>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44604] [Enhancement] Asyncio task decorator to provide interface to define async DAGs (similar to dask's delayed interface)

[Art]

Art  added the comment:

FYI - I've added created a new pypi project for this called `aiodag`. 

pip install aiodag

Github here: https://github.com/aa1371/aiodag
Pypi here: https://pypi.org/project/aiodag/

I would appreciate it if you could find some time take a look just at the 
readme on the github page and provide any thoughts. I still think that this 
would be a good fit for asyncio directly. But if still not convinced is there 
any conditions under which you could see this project eventually being folded 
into the library?

Thanks for your time in reviewing this issue

--

___
Python tracker 
<https://bugs.python.org/issue44604>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6518] Enable 'with' statement in ossaudiodev module

[Art Gillespie]

Art Gillespie  added the comment:

Diff attached

* Added the __enter__ and __exit__ methods to the ossaudio object.
* Updated tests so they pass (attempt to access ossaudio.closed throws
AttributeError instead of TypeError)

My first patch.  Please let me know if I did anything terrifically
stupid. :-)

--
keywords: +patch
nosy: +agillesp
Added file: http://bugs.python.org/file14767/issue6518.diff

___
Python tracker 
<http://bugs.python.org/issue6518>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6759] zipfile.ZipExtFile.read() is missing universal newline support

[Art Gillespie]

Art Gillespie  added the comment:

Patch for both zipfile.py and test_zipfile.py attached.

* The universal newline logic is now in read instead of readline.
* UniversalNewlineTests.read_test changed to check for \n rather than
unchanged eol.

--
keywords: +patch
nosy: +agillesp
type:  -> behavior
Added file: http://bugs.python.org/file14772/issue6759_1.diff

___
Python tracker 
<http://bugs.python.org/issue6759>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6759] zipfile.ZipExtFile.read() is missing universal newline support

[Art Gillespie]

Art Gillespie  added the comment:

Hi Ryan,

Thanks for the feedback.

I've attached a new patch that fixes the read(nbytes) behavior--It will
now always return the requested number of bytes regardless of newline
replacement.  There's now a unit test for this as well.

I also added the newlines attribute per PEP 278 and a corresponding test.

I'm not sure I understood David's comment that read_test needed to be
turned into a unit test:  it's called by several of the unit tests
(test_read_stored, test_read_deflated, et. al.) in that class.

--
Added file: http://bugs.python.org/file14784/issue6759_2.diff

___
Python tracker 
<http://bugs.python.org/issue6759>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6759] zipfile.ZipExtFile.read() is missing universal newline support

[Art Gillespie]

Art Gillespie  added the comment:

Hi David,

Thanks for the review.  Patch attached.

(1)  I've moved that comment to the check's new location.

(2)  Fixed the bug and added tests for only one separator.  Also added
test data and tests for mixed eol files.

(3)  I changed this so that when the file is opened with universal
newline support, read(size) makes multiple calls to _do_read until size
bytes are read or EOF is reached.

--
Added file: http://bugs.python.org/file14785/issue6759_3.diff

___
Python tracker 
<http://bugs.python.org/issue6759>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6518] Enable 'with' statement in ossaudiodev module

[Art Gillespie]

Art Gillespie  added the comment:

Hi Victor,

I copied both the return NULL behavior in oss_exit and the oss_self
naming from Objects/fileobject.c:

http://paste.pocoo.org/show/136451/

Should they be changed there as well?

--

___
Python tracker 
<http://bugs.python.org/issue6518>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6759] zipfile.ZipExtFile.read() is missing universal newline support

[Art Gillespie]

Art Gillespie  added the comment:

Just found another bug in the code that sets the newlines attribute. 
Please disregard issue6759_3.diff

--

___
Python tracker 
<http://bugs.python.org/issue6759>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6759] zipfile.ZipExtFile.read() is missing universal newline support

[Art Gillespie]

Art Gillespie  added the comment:

Latest patch attached.

* Fixed the code that populates the newlines attribute.  I think I've
covered all the cases...

* Found another deviation from file object behavior in this module: 
Calling read with a negative size parameter does not always return the
remainder of the file as described in
http://docs.python.org/library/stdtypes.html#file.read  I went ahead and
fixed this--please let me know if I should open a separate issue and
submit a separate patch.

* Added more tests for mixed eol files, calling read with a negative
size parameter, reading a file with only crlfs

--
Added file: http://bugs.python.org/file14787/issue6759_4.diff

___
Python tracker 
<http://bugs.python.org/issue6759>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5148] gzip.open breaks with 'U' flag

[Art Gillespie]

Art Gillespie  added the comment:

The problem appears to be that the gzip module simply doesn't support
universal newlines yet.

I'm currently working on the zipfile module's universal newline support
(issue6759) so if nobody else is working on this, I'll do it.

I'm not sure if file object's open() behavior when presented with 'rUb'
is correct or not.

>>> f = open("test.txt", "w").write("blah\r\nblah\rblah\nblah\r\n")
>>> f = open("test.txt", "rUb")
>>> f.read()
'blah\nblah\nblah\nblah\n'

Since 'U' and 'b' are conceptually mutually exclusive on platforms where
'b' matters, I can see this being confusing.

--
nosy: +agillesp

___
Python tracker 
<http://bugs.python.org/issue5148>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com