Re: [Twisted-Python] reactor for Linux io_uring

2021-03-13 Thread Tobias Oberstein
Am 13.03.21 um 01:01 schrieb Ian Haywood: On 12/03/2021 8:29 pm, Tobias Oberstein wrote: since with io_uring the queue/ring to append to never blocks for writes, that buffering in twisted would lead to double buffering (as the ring already buffers) then, io_uring follows a completion IO model

Re: [Twisted-Python] reactor for Linux io_uring

2021-03-12 Thread Ian Haywood
On 12/03/2021 8:29 pm, Tobias Oberstein wrote: since with io_uring the queue/ring to append to never blocks for writes, that buffering in twisted would lead to double buffering (as the ring already buffers) then, io_uring follows a completion IO model (as IOCP). quite different from the "read

Re: [Twisted-Python] reactor for Linux io_uring

2021-03-12 Thread Tobias Oberstein
interesting. thanks! Am 12.03.21 um 09:59 schrieb Ian Haywood: On 12/03/2021 7:21 pm, Tobias Oberstein wrote: sorry, I missed the beginning of the thread / discussion: are you working on a new, full Twisted reactor on top of io_uring? and async filesystem is "just" one part / additional effor

Re: [Twisted-Python] reactor for Linux io_uring

2021-03-12 Thread Ian Haywood
On 12/03/2021 7:21 pm, Tobias Oberstein wrote: sorry, I missed the beginning of the thread / discussion: are you working on a new, full Twisted reactor on top of io_uring? and async filesystem is "just" one part / additional effort in that? the advantage of io_uring is being able to combine fi

Re: [Twisted-Python] reactor for Linux io_uring

2021-03-12 Thread Tobias Oberstein
sorry, I missed the beginning of the thread / discussion: are you working on a new, full Twisted reactor on top of io_uring? and async filesystem is "just" one part / additional effort in that? Am 12.03.21 um 05:42 schrieb Ian Haywood: On 11/01/2021 8:26 pm, Glyph wrote: seems somewhat irrelev

Re: [Twisted-Python] reactor for Linux io_uring

2021-03-11 Thread Ian Haywood
On 11/01/2021 8:26 pm, Glyph wrote: seems somewhat irrelevant to the "asynchronous filesystem" part of this PR - do you think you could do a smaller version of this which decouples it from smb and ctypes? but yes both can be spun out easily, so it's just the interface and a plain portable threa

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-11 Thread Glyph
> On Jan 10, 2021, at 4:42 PM, Ian Haywood wrote: > > > > On 8/01/2021 7:23 am, Glyph wrote: >> >> The mess of ctypes stuff > unclear what you mean: either the aio implementation or statx.py > Both, really, but I was mainly talking about statx.py. >> seems somewhat irrelevant to the "async

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-10 Thread Ian Haywood
On 8/01/2021 7:23 am, Glyph wrote: The mess of ctypes stuff unclear what you mean: either the aio implementation or statx.py seems somewhat irrelevant to the "asynchronous filesystem" part of this PR - do you think you could do a smaller version of this which decouples it from smb and ctyp

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-07 Thread Glyph
> On Jan 7, 2021, at 3:24 AM, Ian Haywood wrote: > > For async file I/O my plan would be to export a new IFilesystem (which is > closely based on conch.interfaces.ISFTPServer) apps would be have to be > written to use it, conch could with minimal tweaking, and unsurprisingly the > SMB serve

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-07 Thread Ian Haywood
On 6/01/2021 1:04 am, Adi Roiban wrote: On Tue, 5 Jan 2021 at 13:44, Jean-Paul Calderone mailto:exar...@twistedmatrix.com>> wrote: On Tue, Jan 5, 2021 at 6:49 AM Barry Scott mailto:barry.sc...@forcepoint.com>> wrote: What threads? Why do you call out file FDs different fro

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-05 Thread Ian Haywood
over engineered but don't exactly break new ground in a CS sense. On 6/01/2021 11:35 am, Ian Haywood wrote: Proprietary protocols like SMB tend to be over-eng On 4/01/2021 4:41 pm, Glyph wrote: I suspect that this may require somewhat less... cognitive surface area than your other contribution

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-05 Thread Ian Haywood
Proprietary protocols like SMB tend to be over-eng On 4/01/2021 4:41 pm, Glyph wrote: I suspect that this may require somewhat less... cognitive surface area than your other contributions :). And hey, we have a vaccine now, which means that maybe things will go back to normal or close enough

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-05 Thread Adi Roiban
On Tue, 5 Jan 2021 at 13:44, Jean-Paul Calderone wrote: > On Tue, Jan 5, 2021 at 6:49 AM Barry Scott > wrote: > >> On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote: >> > In investigating async file I/O I came across this. In a nutshell it's >> > the new epoll() >> >> > It's marginally mo

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-05 Thread Kyle Altendorf
On 2021-01-04 10:15, Barry Scott wrote: On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote: If people think an IoUringReactor is worthwhile I'll open a ticket and make a start. I'm guessing that you will need to write a Python extension to get at io_uring or use ctypes. Is that what yo

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-05 Thread Barry Scott
On Tuesday, 5 January 2021 13:38:11 GMT Hynek Schlawack wrote: > >> What's more interesting is that io_uring accepts files as > >> well as network/pipe handles: avoiding the need for threads. > > > > What threads? Why do you call out file FDs different from socket FDs? > > > > The point of io_ur

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-05 Thread Barry Scott
On Tuesday, 5 January 2021 13:31:49 GMT Jean-Paul Calderone wrote: > On Tue, Jan 5, 2021 at 6:49 AM Barry Scott > wrote: > > > On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote: > > > In investigating async file I/O I came across this. In a nutshell it's > > > the new epoll() > > > > > It'

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-05 Thread Hynek Schlawack
>> What's more interesting is that io_uring accepts files as >> well as network/pipe handles: avoiding the need for threads. > > What threads? Why do you call out file FDs different from socket FDs? > > The point of io_uring is to avoid transitions between user and kernel right? > Nothing to do

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-05 Thread Jean-Paul Calderone
On Tue, Jan 5, 2021 at 6:49 AM Barry Scott wrote: > On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote: > > In investigating async file I/O I came across this. In a nutshell it's > > the new epoll() > > > It's marginally more efficient although this is only apparent at very > > high loads.

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-05 Thread Barry Scott
On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote: > In investigating async file I/O I came across this. In a nutshell it's > the new epoll() > It's marginally more efficient although this is only apparent at very > high loads. > What's more interesting is that io_uring accepts files as

Re: [Twisted-Python] reactor for Linux io_uring

2021-01-03 Thread Glyph
I suspect that this may require somewhat less... cognitive surface area than your other contributions :). And hey, we have a vaccine now, which means that maybe things will go back to normal or close enough to it that I'll have enough capacity to get back to it myself :) -g > On Jan 3, 2021,