> Normally the pipe size doesn’t need to be very large if the producer and consumer “costs” are roughly the same. If the producer is costlier than the consumer than a large buffer does nothing. If the consumer is costlier than a large buffer only allow the producer to complete sooner. Otherwise, you only need to make the buffer large enough to overcome > the cost of the system call.
> Eg if the minimal read cost is 6 usec, you need to have the buffer be large enough that the processing costs 600 usecs - to make the IO less than 1%. It is very hard to “process” 2.5gb in 600 usecs. pipe-user-pages-soft is just a limitation, it doesn't means in my case there will have such number of pages allocated. Actually, I just use a file which size 2MB in the benchmark (downloading from another service, and write to the local disk with splice). On Thursday, December 5, 2024 at 10:41:15 AM UTC+8 Robert Engels wrote: > Normally the pipe size doesn’t need to be very large if the producer and > consumer “costs” are roughly the same. If the producer is costlier than the > consumer than a large buffer does nothing. If the consumer is costlier than > a large buffer only allow the producer to complete sooner. Otherwise, you > only need to make the buffer large enough to overcome the cost of the > system call. > > Eg if the minimal read cost is 6 usec, you need to have the buffer be > large enough that the processing costs 600 usecs - to make the IO less than > 1%. It is very hard to “process” 2.5gb in 600 usecs. > > > On Dec 4, 2024, at 8:28 PM, Chao Zhang <zcha...@gmail.com> wrote: > > > > Hi, > > > >> That sets the upper limit on how much data a kernel pipe can buffer to > 2.5 GiB. That is not reasonable. It is also an upper limit, not the default > size. Also, that kernel parameter only existed, AFAIK, in Linux 2.6.34 > which was released in 2010. Are you really using a Linux kernel that old? > > > > My kernel version is 5.4 and besides this parameter, the pipe-max-size > > is 1MB, and I know that the pipe created by the golang runtime has a > > hard-coded size, which is also 1MB. > > > > Best regards > > Chao Zhang > > > > https://github.com/tokers > > > >> On Thu, Dec 5, 2024 at 6:57 AM Kurtis Rader <kra...@skepticism.us> > wrote: > >> > >>> On Wed, Dec 4, 2024 at 12:33 AM tokers <zcha...@gmail.com> wrote: > >>> > >>> I noticed (*os.File).ReadFrom has supported splice(2) since go/1.21, > and I'm trying to introduce it to my project. But the performance is not as > expected, which is slower than the normal io.Copy (buffer size = 1MB). Does > anyone who knows the reason? > >>> > >>> For using splice(2), I tweaked the parameters: > >>> > >>> echo 655360 > /proc/sys/fs/pipe-user-pages-soft > >> > >> > >> That sets the upper limit on how much data a kernel pipe can buffer to > 2.5 GiB. That is not reasonable. It is also an upper limit, not the default > size. Also, that kernel parameter only existed, AFAIK, in Linux 2.6.34 > which was released in 2010. Are you really using a Linux kernel that old? > >> > >> -- > >> Kurtis Rader > >> Caretaker of the exceptional canines Junior and Hank > > > > -- > > You received this message because you are subscribed to the Google > Groups "golang-nuts" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to golang-nuts...@googlegroups.com. > > To view this discussion visit > https://groups.google.com/d/msgid/golang-nuts/CAPr95Bih_tnPc8a3cUiJNE6%3DiVbg61WYh50x5vGqr5JgYqbLPA%40mail.gmail.com > . > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/75579fa6-93ff-472c-8448-b4c746fe4198n%40googlegroups.com.
