Thank you Rajanikanth, Kevin and Egon! I will explore the ideas you guys provided and keep you updated.
Best Regards, Chun On Sunday, May 28, 2017 at 2:25:03 AM UTC-4, Egon wrote: > > > > On Saturday, 27 May 2017 14:05:11 UTC+3, Chun Zhang wrote: >> >> Thanks Kevin and Egon! >> >> With a few experiments, I found that the logging, even to a file, is >> quite time consuming, so turning off logging helps, resulting in >> 500Mbps-ish no drop rate; however, still not even close to Gbps. >> >> Then I turned on both lazy and nocopy option in the decoding option, the >> lazy options seems to help. I got something close to 700Mbps, where the >> sender's limit is reached. >> >> Given that said, the program does not nothing but receiving the packet at >> this moment. Any actual processing of the packet in the same thread >> significantly hurt the rate. Besides spinning multiple thread to handle the >> actual work, anything else in the gopacket land that can be done? >> > > Profile your code. :) > > >> >> Thanks again! >> Chun >> >> >> >> On Saturday, May 27, 2017 at 5:24:13 AM UTC-4, Kevin Conway wrote: >>> >>> > can only handle up to 250Mbps-ish traffic >>> >>> I'm not familiar with gopacket, but I have seen multiple occasions where >>> logging to a file or stdout became a bottleneck. Your code snippet is >>> logging on every packet which seems excessive. Try logging with less >>> frequency and, if using stdout, consider using a log destination with >>> different buffering characteristics like a file or syslog over UDP. >>> >>> On Fri, May 26, 2017 at 3:59 PM Egon <egon...@gmail.com> wrote: >>> >>>> On Friday, 26 May 2017 20:51:55 UTC+3, Chun Zhang wrote: >>>>> >>>>> Good point. >>>>> as a comparison: tcpdump -w /dev/null can handle up to 750Mbps, where >>>>> sending machine's speed limit reached. I think it should be able to >>>>> handle >>>>> line rate. >>>>> >>>>> Are those two packages lighter/faster than gopacket? >>>>> >>>> >>>> Nevermind, just noticed... gopacket/pcap is a fork of akrennmair/gopcap >>>> >>>> Anyways, to get more information on what is taking time in your program >>>> see https://blog.golang.org/profiling-go-programs >>>> >>>> Maybe try something like this: >>>> >>>> handle, err := pcap.OpenLive(device, snapshot_len, promiscuous, timeout) >>>> >>>> // ... >>>> for { >>>> data, ci, err := handle.ZeroCopyReadPacketData() >>>> // ... >>>> >>>> >>>> This should remove allocations from critical path. >>>> >>>> *PS: code untested and may contain typos :P* >>>> >>>> >>>>> >>>>> >>>>> Thanks, >>>>> Chun >>>>> >>>>> On Friday, May 26, 2017 at 12:37:55 PM UTC-4, Egon wrote: >>>>>> >>>>>> As a baseline measurement I suggest writing the same code in C; this >>>>>> shows how much your VM / config / machine can handle. >>>>>> >>>>>> With gopacket -- use src.NextPacket instead of Packets. >>>>>> >>>>>> There are also: https://github.com/akrennmair/gopcap and >>>>>> https://github.com/miekg/pcap >>>>>> >>>>>> + Egon >>>>>> >>>>>> On Friday, 26 May 2017 19:01:20 UTC+3, Chun Zhang wrote: >>>>>>> >>>>>>> Hi, All, >>>>>>> >>>>>>> I am trying to write a small program to handle packets coming from a >>>>>>> GigE wire. The testing code snip is as below. >>>>>>> >>>>>>> The problem I am facing is that this is extremely slow, can only >>>>>>> handle up to 250Mbps-ish traffic with normal ipv4 sized packets, >>>>>>> anything >>>>>>> above that resulting significant packet drop. Note that I am doing >>>>>>> nothing >>>>>>> with the packet at this moment. If I try to do any packet processing, >>>>>>> then >>>>>>> apparently it gets slower. >>>>>>> >>>>>>> Has anybody measured the efficiency of the gopacket package? Is >>>>>>> there any other faster alternatives? >>>>>>> >>>>>>> PS: the host machine is an ubuntu VM with 12-core and 12G memory, >>>>>>> but looks only 2 cores are used for this program. >>>>>>> >>>>>>> Thanks, >>>>>>> Chun >>>>>>> >>>>>>> >>>>>>> >>>>>>> // Open device >>>>>>> handle, err = pcap.OpenLive(device, snapshot_len, promiscuous, timeout) >>>>>>> if err == nil { >>>>>>> Info.Println("Open interface ", device, "successfully") >>>>>>> >>>>>>> } >>>>>>> defer handle.Close() >>>>>>> >>>>>>> >>>>>>> //fmt.Println("In the deafult reading case ", time.Now()) >>>>>>> // Use the handle as a packet source to process all packets >>>>>>> packetSource := gopacket.NewPacketSource(handle, >>>>>>> handle.LinkType()) >>>>>>> Info.Println("pcketsourc is ", packetSource, time.Now()) >>>>>>> for packet := range packetSource.Packets() { >>>>>>> >>>>>>> Debug.Println("-------------------------------------------------------------------") >>>>>>> count++ >>>>>>> Warning.Println("packet count ", count) >>>>>>> >>>>>>> // write to a pcap for testing >>>>>>> /*err = w.WritePacket(packet.Metadata().CaptureInfo, >>>>>>> packet.Data()) >>>>>>> if err != nil { >>>>>>> fmt.Println(err) >>>>>>> }*/ >>>>>>> >>>>>>> continue >>>>>>> >>>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "golang-nuts" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to golang-nuts...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
