Normally the pipe size doesn’t need to be very large if the producer and consumer “costs” are roughly the same. If the producer is costlier than the consumer than a large buffer does nothing. If the consumer is costlier than a large buffer only allow the producer to complete sooner. Otherwise, you only need to make the buffer large enough to overcome the cost of the system call.
Eg if the minimal read cost is 6 usec, you need to have the buffer be large enough that the processing costs 600 usecs - to make the IO less than 1%. It is very hard to “process” 2.5gb in 600 usecs. > On Dec 4, 2024, at 8:28 PM, Chao Zhang <zchao1...@gmail.com> wrote: > > Hi, > >> That sets the upper limit on how much data a kernel pipe can buffer to 2.5 >> GiB. That is not reasonable. It is also an upper limit, not the default >> size. Also, that kernel parameter only existed, AFAIK, in Linux 2.6.34 which >> was released in 2010. Are you really using a Linux kernel that old? > > My kernel version is 5.4 and besides this parameter, the pipe-max-size > is 1MB, and I know that the pipe created by the golang runtime has a > hard-coded size, which is also 1MB. > > Best regards > Chao Zhang > > https://github.com/tokers > >> On Thu, Dec 5, 2024 at 6:57 AM Kurtis Rader <kra...@skepticism.us> wrote: >> >>> On Wed, Dec 4, 2024 at 12:33 AM tokers <zchao1...@gmail.com> wrote: >>> >>> I noticed (*os.File).ReadFrom has supported splice(2) since go/1.21, and >>> I'm trying to introduce it to my project. But the performance is not as >>> expected, which is slower than the normal io.Copy (buffer size = 1MB). Does >>> anyone who knows the reason? >>> >>> For using splice(2), I tweaked the parameters: >>> >>> echo 655360 > /proc/sys/fs/pipe-user-pages-soft >> >> >> That sets the upper limit on how much data a kernel pipe can buffer to 2.5 >> GiB. That is not reasonable. It is also an upper limit, not the default >> size. Also, that kernel parameter only existed, AFAIK, in Linux 2.6.34 which >> was released in 2010. Are you really using a Linux kernel that old? >> >> -- >> Kurtis Rader >> Caretaker of the exceptional canines Junior and Hank > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion visit > https://groups.google.com/d/msgid/golang-nuts/CAPr95Bih_tnPc8a3cUiJNE6%3DiVbg61WYh50x5vGqr5JgYqbLPA%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/92AFDD20-3E80-4659-A370-5A73C3556054%40ix.netcom.com.
