[issue17552] socket.sendfile()

Giampaolo Rodola' added the comment: Given the opinions expressed so far I: - got rid of the "blocksize" parameter - got rid of the "use_fallback" parameter - added a "count" parameter - used os.fstat() to figure out the total file size and passed it directly to sendfile() I'm attaching socket-

[issue17552] socket.sendfile()

akira added the comment: > I'm confused. Why is "blocksize" necessary at all? My guess, it may be used to implement socket.send()-based fallback. Its meaning could be the same as *length* parameter in shutil.copyfileobj The fallback is useful if os.sendfile doesn't exists or it doesn't accept

[issue17552] socket.sendfile()

Antoine Pitrou added the comment: > I'm -1 about adding "count" *and* "blocksize" parameters. "blocksize" > alone > is good enough IMO and considering what I've just described it > is a better name than "count". I'm confused. Why is "blocksize" necessary at all? > using os.path.getsize(file.na

[issue17552] socket.sendfile()

akira added the comment: count and blocksize are completely different. *count* specifies how many bytes at most socket.sendfile should sent overall. It may change the result i.e., it may not be necessary that the file is read until EOF. It has the same meaning as *nbytes* parameter for os.sen

[issue17552] socket.sendfile()

Giampaolo Rodola' added the comment: > [...] I'd like a parameter for the offset, and another one for the > number of bytes to send. > To sum up, I think there's a fundamental confusion between blocksize > and count in this API. Ah OK, I see what you mean now. If seems we didn't understand each

[issue17552] socket.sendfile()

akira added the comment: use_fallback parameter is mostly a debugging tool. If it helps to avoid the indecision; I would side with neologix' remarks and also suggest to drop the use_fallback parameter. It seems the patch assumes *offset == nbytes_sent* that is false in general e.g., if offset >

[issue17552] socket.sendfile()

Charles-François Natali added the comment: >> A useful parameter instead would be to support sending only part of the file, >> so adding a count argument. > > Have you read my patch? This is already provided by the "offset" parameter. Of course I read your patch ;-) I mean I'd like a parameter f

[issue17552] socket.sendfile()

Giampaolo Rodola' added the comment: I think asyncio would be better off using os.sendfile() / TransmitFile directly, in fact the current patch explicitly does not support non-blocking sockets (I couldn't see any sane approach to do that). Here's an example of how os.sendfile() should be used i

[issue17552] socket.sendfile()

Guido van Rossum added the comment: Can you also think about how this would be wrapped in asyncio? -- nosy: +gvanrossum ___ Python tracker ___ ___

[issue17552] socket.sendfile()

Changes by Yury Selivanov : -- nosy: +yselivanov ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.p

[issue17552] socket.sendfile()

Giampaolo Rodola' added the comment: Considering the current indecision about certain design aspects I started a discussion on python-ideas: https://mail.python.org/pipermail/python-ideas/2014-April/027752.html -- ___ Python tracker

[issue17552] socket.sendfile()

akira added the comment: > I really don't like the use_fallback argument .. I initially also thought so. But I've suggested the parameter to replace `(was_os_sendfile_used, os_sendfile_error)` returned value as a *trade off* between a slight complexity in the interface vs. allowing to detect p

[issue17552] socket.sendfile()

Giampaolo Rodola' added the comment: Note: my example about limiting the transfer speed does not really apply 'cause as this stands right now it cannot be used with non-blocking sockets. Other arguments do though and I hope it's clear that we need "blocksize". -- _

[issue17552] socket.sendfile()

Giampaolo Rodola' added the comment: > 1) I really don't like the use_fallback argument > Apart from complicating the prototype, what do this bring? My initial thought was that the user might want to know *why* a file cannot be sent by using the fastest method and hence wants to see the origina

[issue17552] socket.sendfile()

Charles-François Natali added the comment: 1) I really don't like the use_fallback argument: as a user, I don't care if it's using sendfile/splice/whatever WIndows uses. I view this as a channel transfer (like Java's http://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html#tran

[issue17552] socket.sendfile()

Giampaolo Rodola' added the comment: Yet another patch fixing some problems on Windows. Hopefully this should be the last one as for what concerns the POSIX systems. As such I would kindly ask for a review and/or further suggestions. -- Added file: http://bugs.python.org/file35004/sendf

[issue17552] socket.sendfile()

Changes by Giampaolo Rodola' : -- nosy: +josiah.carlson ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:/

[issue17552] socket.sendfile()

Giampaolo Rodola' added the comment: > Instead of returning [...] you could specify `no_fallback=False` that > could be set to `True` to assert that the fallback is not used > [...] and return the number of bytes sent. Good idea, thanks, that is much better indeed. Updated patch is in attachmen

[issue17552] socket.sendfile()

akira added the comment: Should socket.sendfile() always return number of bytes sent because file.tell() may be changed by something else that uses the same file descriptor? What happens if the file grows? Instead of returning `(was_os_sendfile_used, os_sendfile_error)`, you could specify `no

[issue17552] socket.sendfile()

Josh Rosenberg added the comment: For TransmitFile support, the Windows function to turn an integer file descriptor into a WinAPI file HANDLE should be _get_osfhandle: http://msdn.microsoft.com/en-us/library/ks2530z6.aspx -- nosy: +josh.rosenberg __

[issue17552] socket.sendfile()

Giampaolo Rodola' added the comment: Attached is a simple benchmark script transmitting a 100MB file. On my Linux box sendfile() is almost twice as fast as send(): send() real 0.0613s user 0.0100s sys 0.0900s total0.1000s sendfile() real 0.0318s user 0.s sys 0

[issue17552] socket.sendfile()

Changes by Giampaolo Rodola' : -- versions: +Python 3.5 -Python 3.4 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue17552] socket.sendfile()

Giampaolo Rodola' added the comment: New patch in attachment. Changes: - docs - replaced select() / poll() with the new selectors module - file position is always updated both on return and on error; this means file.tell() is the designated way to know how many bytes were sent - replaced sen

[issue17552] socket.sendfile()

Changes by Giampaolo Rodola' : Added file: http://bugs.python.org/file29734/socket-sendfile2.patch ___ Python tracker ___ ___ Python-bugs-list

[issue17552] socket.sendfile()

Changes by Giampaolo Rodola' : Removed file: http://bugs.python.org/file29733/socket-sendfile2.patch ___ Python tracker ___ ___ Python-bugs-li

[issue17552] socket.sendfile()

Changes by Giampaolo Rodola' : Added file: http://bugs.python.org/file29733/socket-sendfile2.patch ___ Python tracker ___ ___ Python-bugs-list

[issue17552] socket.sendfile()

Giampaolo Rodola' added the comment: New patch in attachment includes a new 'offset' parameter, new tests and also update file offset on return or in case of error so that file.tell() can be used to tell how many bytes were transmitted at any time. This way we'll avoid using a custom exception.

[issue17552] socket.sendfile()

Changes by Giampaolo Rodola' : -- title: create_server -> socket.sendfile() ___ Python tracker ___ ___ Python-bugs-list mailing list U

[issue17552] socket.sendfile()

Changes by Andrew Svetlov : -- nosy: +asvetlov ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue17552] socket.sendfile()

Changes by Ross Lagerwall : -- nosy: +rosslagerwall ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail

[issue17552] socket.sendfile()

Giampaolo Rodola' added the comment: > I don't understand the point of the second member in the tuple The 'exception' member can be useful to know the reason why sendfile() failed and send() was used as fallback. > the timeout logic should be fixed so that the total operation > time doesn't e

[issue17552] socket.sendfile()

Changes by Christian Heimes : -- nosy: +christian.heimes ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:/

[issue17552] socket.sendfile()

Charles-François Natali added the comment: - I don't understand the "running out of FDs" thing. select() is limited to FDs less than FD_SETSIZE, but I don't see how you could get EMFILE (select() will return a ValueError) - is there any platform with sendfile() which doesn't support poll()? -

[issue17552] socket.sendfile()

Antoine Pitrou added the comment: A couple of comments: - I don't understand the point of the second member in the tuple - the timeout logic should be fixed so that the total operation time doesn't exceed the timeout, rather than each iteration (in other words, a deadline should be computed at

[issue17552] socket.sendfile()

New submission from Giampaolo Rodola': This is based on progress made in issue 13564 and aims to provide a new sendfile() method for the socket class taking advantage of high-performance "zero-copy" os.sendfile() available on most POSIX platforms. A wrapper on top of os.sendfile() appears to be