Thank you Kevin for spotting the unnecessary memory allocation! Took the example code naively and didn't think through :) That does make an impact to the app and the throughput is increased by another 30Mbps to 150Mbps - ish. As a benchmark, a similar C++ version of the app is able to handle 250Mbps.
I am digging into the profiling and try to optimize further. Best Regards, Chun On Friday, June 9, 2017 at 1:31:18 PM UTC-4, Kevin Conway wrote: > > On first appearance, the article you linked to uses some patterns that are > different from the same documentation in gopacket: > https://godoc.org/github.com/google/gopacket#hdr-Fast_Decoding_With_DecodingLayerParser > > . Namely, the official docs suggest reusing the parser and results > allocations while the third party docs demonstrate reallocating these > components on each packet. It's worth double checking that you aren't > performing unnecessary allocations due to bad example code. > > Also, I recommend reading through > https://blog.golang.org/profiling-go-programs if you haven't already. It > gives some good examples of drilling down into smaller components to find > more granular code bottlenecks. > > On Fri, Jun 9, 2017, 09:47 Chun Zhang <chun...@gmail.com <javascript:>> > wrote: > >> Hi, All, >> >> Update on this issue. Based on the suggestion I got earlier, I dedicated >> one thread, which is locked to a os thread to handle packet receiving, this >> thread then put the received packet on a buffered channel/queue. Without >> doing any extra work, this thread is able to take packets up to >> 800Mbps-ish, which is the limit of the sender. >> >> 12 goroutines are then kicked off to take items from this queue and >> distribute them to other work queues for further processing. The >> distributing thread parses the packet to a gopacket.Packet interface and >> hashes based on the ip address etc. Even though I have used the faster >> version, the parse routine takes a LOT of the cpu power. By profiling the >> program, it seems the parsing takes 1/3 of whole time. The whole app is >> then limited to roughly 120Mbps, aka, 50kpps. >> >> The decoding routine I am using is pretty much like the example here >> Decoding Packets Faster >> >> http://www.devdungeon.com/content/packet-capture-injection-and-analysis-gopacket >> >> I am wondering what further optimization can I do to speed this up? >> >> Thanks, >> Chun >> >> >> >> >> >> On Tuesday, May 30, 2017 at 10:50:50 AM UTC-4, Chun Zhang wrote: >>> >>> Thank you Rajanikanth, Kevin and Egon! I will explore the ideas you >>> guys provided and keep you updated. >>> >>> Best Regards, >>> Chun >>> >>> On Sunday, May 28, 2017 at 2:25:03 AM UTC-4, Egon wrote: >>>> >>>> >>>> >>>> On Saturday, 27 May 2017 14:05:11 UTC+3, Chun Zhang wrote: >>>>> >>>>> Thanks Kevin and Egon! >>>>> >>>>> With a few experiments, I found that the logging, even to a file, is >>>>> quite time consuming, so turning off logging helps, resulting in >>>>> 500Mbps-ish no drop rate; however, still not even close to Gbps. >>>>> >>>>> Then I turned on both lazy and nocopy option in the decoding option, >>>>> the lazy options seems to help. I got something close to 700Mbps, where >>>>> the >>>>> sender's limit is reached. >>>>> >>>>> Given that said, the program does not nothing but receiving the packet >>>>> at this moment. Any actual processing of the packet in the same thread >>>>> significantly hurt the rate. Besides spinning multiple thread to handle >>>>> the >>>>> actual work, anything else in the gopacket land that can be done? >>>>> >>>> >>>> Profile your code. :) >>>> >>>> >>>>> >>>>> Thanks again! >>>>> Chun >>>>> >>>>> >>>>> >>>>> On Saturday, May 27, 2017 at 5:24:13 AM UTC-4, Kevin Conway wrote: >>>>>> >>>>>> > can only handle up to 250Mbps-ish traffic >>>>>> >>>>>> I'm not familiar with gopacket, but I have seen multiple occasions >>>>>> where logging to a file or stdout became a bottleneck. Your code snippet >>>>>> is >>>>>> logging on every packet which seems excessive. Try logging with less >>>>>> frequency and, if using stdout, consider using a log destination with >>>>>> different buffering characteristics like a file or syslog over UDP. >>>>>> >>>>>> On Fri, May 26, 2017 at 3:59 PM Egon <egon...@gmail.com> wrote: >>>>>> >>>>>>> On Friday, 26 May 2017 20:51:55 UTC+3, Chun Zhang wrote: >>>>>>>> >>>>>>>> Good point. >>>>>>>> as a comparison: tcpdump -w /dev/null can handle up to 750Mbps, >>>>>>>> where sending machine's speed limit reached. I think it should be >>>>>>>> able to >>>>>>>> handle line rate. >>>>>>>> >>>>>>>> Are those two packages lighter/faster than gopacket? >>>>>>>> >>>>>>> >>>>>>> Nevermind, just noticed... gopacket/pcap is a fork >>>>>>> of akrennmair/gopcap >>>>>>> >>>>>>> Anyways, to get more information on what is taking time in your >>>>>>> program see https://blog.golang.org/profiling-go-programs >>>>>>> >>>>>>> Maybe try something like this: >>>>>>> >>>>>>> handle, err := pcap.OpenLive(device, snapshot_len, promiscuous, >>>>>>> timeout) >>>>>>> >>>>>>> // ... >>>>>>> for { >>>>>>> data, ci, err := handle.ZeroCopyReadPacketData() >>>>>>> // ... >>>>>>> >>>>>>> >>>>>>> This should remove allocations from critical path. >>>>>>> >>>>>>> *PS: code untested and may contain typos :P* >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Chun >>>>>>>> >>>>>>>> On Friday, May 26, 2017 at 12:37:55 PM UTC-4, Egon wrote: >>>>>>>>> >>>>>>>>> As a baseline measurement I suggest writing the same code in C; >>>>>>>>> this shows how much your VM / config / machine can handle. >>>>>>>>> >>>>>>>>> With gopacket -- use src.NextPacket instead of Packets. >>>>>>>>> >>>>>>>>> There are also: https://github.com/akrennmair/gopcap and >>>>>>>>> https://github.com/miekg/pcap >>>>>>>>> >>>>>>>>> + Egon >>>>>>>>> >>>>>>>>> On Friday, 26 May 2017 19:01:20 UTC+3, Chun Zhang wrote: >>>>>>>>>> >>>>>>>>>> Hi, All, >>>>>>>>>> >>>>>>>>>> I am trying to write a small program to handle packets coming >>>>>>>>>> from a GigE wire. The testing code snip is as below. >>>>>>>>>> >>>>>>>>>> The problem I am facing is that this is extremely slow, can only >>>>>>>>>> handle up to 250Mbps-ish traffic with normal ipv4 sized packets, >>>>>>>>>> anything >>>>>>>>>> above that resulting significant packet drop. Note that I am doing >>>>>>>>>> nothing >>>>>>>>>> with the packet at this moment. If I try to do any packet >>>>>>>>>> processing, then >>>>>>>>>> apparently it gets slower. >>>>>>>>>> >>>>>>>>>> Has anybody measured the efficiency of the gopacket package? Is >>>>>>>>>> there any other faster alternatives? >>>>>>>>>> >>>>>>>>>> PS: the host machine is an ubuntu VM with 12-core and 12G memory, >>>>>>>>>> but looks only 2 cores are used for this program. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Chun >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> // Open device >>>>>>>>>> handle, err = pcap.OpenLive(device, snapshot_len, promiscuous, >>>>>>>>>> timeout) >>>>>>>>>> if err == nil { >>>>>>>>>> Info.Println("Open interface ", device, "successfully") >>>>>>>>>> >>>>>>>>>> } >>>>>>>>>> defer handle.Close() >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> //fmt.Println("In the deafult reading case ", time.Now()) >>>>>>>>>> // Use the handle as a packet source to process all packets >>>>>>>>>> packetSource := gopacket.NewPacketSource(handle, >>>>>>>>>> handle.LinkType()) >>>>>>>>>> Info.Println("pcketsourc is ", packetSource, time.Now()) >>>>>>>>>> for packet := range packetSource.Packets() { >>>>>>>>>> >>>>>>>>>> Debug.Println("-------------------------------------------------------------------") >>>>>>>>>> count++ >>>>>>>>>> Warning.Println("packet count ", count) >>>>>>>>>> >>>>>>>>>> // write to a pcap for testing >>>>>>>>>> /*err = w.WritePacket(packet.Metadata().CaptureInfo, >>>>>>>>>> packet.Data()) >>>>>>>>>> if err != nil { >>>>>>>>>> fmt.Println(err) >>>>>>>>>> }*/ >>>>>>>>>> >>>>>>>>>> continue >>>>>>>>>> >>>>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "golang-nuts" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to golang-nuts...@googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
