On Mon, Apr 27, 2020 at 6:59 PM Liam <networkimp...@gmail.com> wrote:
>
> On Monday, April 27, 2020 at 5:56:52 PM UTC-7, Ian Lance Taylor wrote:
>>
>> On Mon, Apr 27, 2020 at 5:10 PM Liam <networ...@gmail.com> wrote:
>> >
>> > On Monday, April 27, 2020 at 4:22:41 PM UTC-7, Ian Lance Taylor wrote:
>> >>
>> >> On Sun, Apr 26, 2020 at 4:55 PM Liam <networ...@gmail.com> wrote:
>> >> >
>> >> > During an io.Copy() where the Writer is a TCPConn and the Reader is a
>> >> > 200K disk file, my code may concurrently Write() on the same TCPConn.
>> >> >
>> >> > I see the result of the Write() inserted into the result of the
>> >> > io.Copy(). I had the impression that was impossible, but I must be
>> >> > mistaken, as the sendfile(2) docs read:
>> >> >
>> >> > Note that a successful call to sendfile() may write fewer bytes than
>> >> > requested; the caller should be prepared to retry the call if there
>> >> > were unsent bytes.
>> >> >
>> >> > Could someone confirm that one must indeed synchronize concurrent use
>> >> > of tcpConn.Write() and io.Copy(tcpConn, file)?
>> >>
>> >> Synchronization should not be required. internal/poll.Sendfile
>> >> acquires a write lock on dstFD, which is the TCP socket. That should
>> >> ensure that the contents of an ordinary Write (which also acquires a
>> >> write lock) should not interleave with the sendfile data.
>> >>
>> >> That said, if the sendfile system call cannot be used for whatever
>> >> reason, the net package will fall back on doing ordinary Read and
>> >> Write calls. And those Write calls can be interleaved with other
>> >> Write calls done by a different goroutine. I think that is probably
>> >> permitted, in that io.Copy doesn't promise to not interleave with
>> >> simultaneous Write calls on the destination.
>> >>
>> >> So in the general case you should indeed use your own locking to avoid
>> >> interleaving between io.Copy and a concurrent Write.
>> >
>> >
>> > Thanks for the details. Where could I add a Println() to reveal why it
>> > doesn't call poll.Sendfile()?
>> >
>> > I expect this system to use sendfile(2). The file is a normal file on a
>> > local partition (running on a Digital Ocean Droplet).
>> >
>> >
>> > /etc/fstab has:
>> > UUID=[omitted] / ext4 defaults 1 1
>> >
>> >
>> > $ df -h
>> > Filesystem Size Used Avail Use% Mounted on
>> > devtmpfs 981M 0 981M 0% /dev
>> > tmpfs 996M 0 996M 0% /dev/shm
>> > tmpfs 996M 436K 995M 1% /run
>> > tmpfs 996M 0 996M 0% /sys/fs/cgroup
>> > /dev/vda1 59G 5.7G 51G 11% /
>> > tmpfs 200M 0 200M 0% /run/user/0
>>
>>
>> Well, I don't know for that it doesn't use sendfile, I just can't
>> explain the results you're seeing if it does use sendfile.
>>
>> The place to start is to find out why (or whether)
>> internal/poll.Sendfile is returning 0, nil.
>
>
> Found the invocations, wrong letter case...
>
> $ grep -r 'SendFile(' /usr/local/go/src/
> /usr/local/go/src/net/sendfile_windows.go: done, err :=
> poll.SendFile(&fd.pfd, syscall.Handle(f.Fd()), n)
> /usr/local/go/src/net/sendfile_linux.go: written, werr =
> poll.SendFile(&c.pfd, int(fd), remain)
> /usr/local/go/src/net/sendfile_unix_alt.go: written, werr =
> poll.SendFile(&c.pfd, int(fd), pos, remain)
> /usr/local/go/src/internal/poll/sendfile_windows.go:func SendFile(fd *FD, src
> syscall.Handle, n int64) (int64, error) {
> /usr/local/go/src/internal/poll/sendfile_linux.go:func SendFile(dstFD *FD,
> src int, remain int64) (int64, error) {
> /usr/local/go/src/internal/poll/sendfile_solaris.go:func SendFile(dstFD *FD,
> src int, pos, remain int64) (int64, error) {
> /usr/local/go/src/internal/poll/sendfile_bsd.go:func SendFile(dstFD *FD, src
> int, pos, remain int64) (int64, error) {
>
> $ grep -r 'sendFile(' /usr/local/go/src/net/
> /usr/local/go/src/net/sendfile_windows.go:func sendFile(fd *netFD, r
> io.Reader) (written int64, err error, handled bool) {
> /usr/local/go/src/net/sendfile_linux.go:func sendFile(c *netFD, r io.Reader)
> (written int64, err error, handled bool) {
> /usr/local/go/src/net/sendfile_unix_alt.go:func sendFile(c *netFD, r
> io.Reader) (written int64, err error, handled bool) {
> /usr/local/go/src/net/sendfile_stub.go:func sendFile(c *netFD, r io.Reader)
> (n int64, err error, handled bool) {
> /usr/local/go/src/net/tcpsock_posix.go: if n, err, handled := sendFile(c.fd,
> r); handled {
>
> Should I check the 'handled' result of sendFile() in net/tcpsock_posix.go ?
Sure. Should be the same thing as the result of poll.SendFile.
Ian
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVHJVZUtNBu04bxwK6qeHE9-VGJF3J%2B3K90KGzphP%3DaTA%40mail.gmail.com.
