Re: [go-nuts] io.CopyN occasionally fails with unexpected EOF on large ranged reads

[Steven Hartland]

What's the source and target, are they networked if so likely a
network issue.

On Mon, 7 Jul 2025 at 18:12, Kanak Bhatia <kanakbhati...@gmail.com> wrote:

> Hi all,
>
> I'm encountering an intermittent issue while using io.CopyN to copy 129MB
> object from reader to writer.
>
> The problem:
> On some iterations, io.CopyN(dst, src, n) fails with unexpected EOF even
> though the same logic succeeds in most runs. The failure typically happens
> after 125–130 MB and returns ~1MB less data than expected.
>
> type debugReader struct {
>     r          io.Reader
>     totalBytes int64
>     nextLogMB  int64
> }
>
> func (d *debugReader) Read(p []byte) (int, error) {
>     n, err := d.r.Read(p)
>     d.totalBytes += int64(n)
>     if d.totalBytes >= 125*1024*1024 {
>         currentMB := d.totalBytes / (1024 * 1024)
>         if currentMB >= d.nextLogMB {
>             fmt.Printf("🔵 Read %d MB so far\n", currentMB)
>             d.nextLogMB = currentMB + 1
>         }
>     }
>     return n, err
> }
>
> type debugWriter struct {
>     w          io.Writer
>     totalBytes int64
>     nextLogMB  int64
> }
>
> func (d *debugWriter) Write(p []byte) (int, error) {
>     n, err := d.w.Write(p)
>     d.totalBytes += int64(n)
>     if d.totalBytes >= 125*1024*1024 {
>         currentMB := d.totalBytes / (1024 * 1024)
>         if currentMB >= d.nextLogMB {
>             fmt.Printf("🟡 Written %d MB so far\n", currentMB)
>             d.nextLogMB = currentMB + 1
>         }
>     }
>     return n, err
> }
>
> func SafeCopyN(dst *debugWriter, src *debugReader, n int64) (int64, error)
> {
>     fmt.Printf("\n🔁 Starting SafeCopyN for %d bytes\n", n)
>     written, err := io.CopyN(dst, src, n)
>     fmt.Printf("📊 Total bytes read:    %d\n", src.totalBytes)
>     fmt.Printf("📊 Total bytes written: %d\n", dst.totalBytes)
>
>     const allowedDrift = 128 * 1024
>
>     if err != nil {
>         diff := n - written
>         if err == io.ErrUnexpectedEOF && diff <= allowedDrift {
>             fmt.Printf("⚠  Accepting unexpected EOF: copied %d of %d bytes
> (drift: %d bytes)\n", written, n, diff)
>             return written, nil
>         }
>
>         fmt.Printf("❌ CopyN failed: wrote %d of %d bytes, err: %v\n",
> written, n, err)
>         if src.totalBytes != dst.totalBytes {
>             fmt.Printf("📉 Mismatch: %d bytes read but not written\n",
> src.totalBytes-dst.totalBytes)
>         }
>     } else {
>         fmt.Printf("✅ Successfully copied %d bytes\n", written)
>     }
>
>     return written, err
> }
>
>
> ---
>
> Here's a real log from one of the failed runs (test runs 20 times, fails
> 2):
>
>
> 🔁 Starting SafeCopyN for 135264256 bytes
> 🔵 Read 125 MB so far
> 🟡 Written 125 MB so far
> 🔵 Read 126 MB so far
> 🟡 Written 126 MB so far
> 🔵 Read 127 MB so far
> 🟡 Written 127 MB so far
> 📊 Total bytes read:    134215680
> 📊 Total bytes written: 134215680
> ❌ CopyN failed: wrote 134215680 of 135264256 bytes, err: unexpected EOF
> 📉 Mismatch: 0 bytes read but not written
>
> Notes:
>
> This only happens 1 to 2 times out of 20 iteration.
>
> No server-side encryption involved.
> ---
>
> Question:
>
> Has anyone seen something similar? Could this be:
>
> Timeout/internal truncation in io.CopyN?
>
> Should io.CopyN tolerate io.ErrUnexpectedEOF more gracefully in this case?
>
>
> Any insights or guidance appreciated!
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion visit
> https://groups.google.com/d/msgid/golang-nuts/56d2219e-32ab-4950-ab49-83b5fbbfc4fen%40googlegroups.com
> <https://groups.google.com/d/msgid/golang-nuts/56d2219e-32ab-4950-ab49-83b5fbbfc4fen%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/CAA38peb7AUHewQk%2Bb%2Bf3o9uZ1t8DWrmL2xVBo%2Bgsj1x9Y%3Dejjw%40mail.gmail.com.