Hi,
Requesting valuable inputs on this.
Thanks,
Nitish
On Thu, Mar 26, 2020 at 5:08 PM Nitish Saboo <nitish.sabo...@gmail.com>
wrote:
> Hi Tamas,
>
> 1) There is no such option '--heap_inuse'. We have an -*-inuse_space*
> option. Is this what you are talking about?
>
> nsaboo@ubuntu:~/Desktop/memprof$ go tool pprof --inuse_space main
> mem3.prof
> Fetched 1 source profiles out of 2
> File: main
> Build ID: 99b8f2b91a4e037cf4a622aa32f2c1866764e7eb
> Type: inuse_space
> Time: Mar 22, 2020 at 6:32am (PDT)
> Entering interactive mode (type "help" for commands, "o" for options)
> (pprof) o
> call_tree = false
> compact_labels = true
> cumulative = flat //: [cum | flat]
> divide_by = 1
> drop_negative = false
> edgefraction = 0.001
> focus = ""
> granularity = filefunctions //: [addresses |
> filefunctions | files | functions | lines]
> hide = ""
> ignore = ""
> mean = false
> nodecount = -1 //: default
> nodefraction = 0.005
> noinlines = false
> normalize = false
> output = ""
> prune_from = ""
> relative_percentages = false
> sample_index = inuse_space //: [alloc_objects |
> alloc_space | inuse_objects | inuse_space]
> show = ""
> show_from = ""
> tagfocus = ""
> taghide = ""
> tagignore = ""
> tagshow = ""
> trim = true
> trim_path = ""
> unit = minimum
>
> 2) When I don't pass the flag it defaults to *--inuse_space*:
>
> File: main
> Build ID: 99b8f2b91a4e037cf4a622aa32f2c1866764e7eb
> Type: inuse_space
> Time: Mar 22, 2020 at 6:32am (PDT)
> Entering interactive mode (type "help" for commands, "o" for options)
> (pprof) top
> Showing nodes accounting for 3303.21kB, 100% of 3303.21kB total
> Showing top 10 nodes out of 28
> flat flat% sum% cum cum%
> 1762.94kB 53.37% 53.37% 1762.94kB 53.37% runtime/pprof.StartCPUProfile
> 516.01kB 15.62% 68.99% 516.01kB 15.62% runtime/pprof.(*profMap).lookup
> 512.19kB 15.51% 84.50% 512.19kB 15.51% runtime.malg
> 512.08kB 15.50% 100% 512.08kB 15.50%
> crypto/x509/pkix.(*Name).FillFromRDNSequence
> 0 0% 100% 512.08kB 15.50% crypto/tls.(*Conn).Handshake
> 0 0% 100% 512.08kB 15.50%
> crypto/tls.(*Conn).clientHandshake
> 0 0% 100% 512.08kB 15.50%
> crypto/tls.(*Conn).verifyServerCertificate
> 0 0% 100% 512.08kB 15.50%
> crypto/tls.(*clientHandshakeState).doFullHandshake
> 0 0% 100% 512.08kB 15.50%
> crypto/tls.(*clientHandshakeState).handshake
> 0 0% 100% 512.08kB 15.50%
> crypto/x509.(*CertPool).AppendCertsFromPEM
>
> Please correct me if I am missing something here.
>
> Thanks,
> Nitish
>
>
> On Wed, Mar 25, 2020 at 12:16 AM Tamás Gulácsi <tgulacs...@gmail.com>
> wrote:
>
>> You've requested the total allocated space (--alloc_space), not only the
>> heap used (--heap_inuse, or no flag).
>> So that 17GiB is the total allocated size, does NOT include the released!
>>
>> 2020. március 24., kedd 15:16:46 UTC+1 időpontban Nitish Saboo a
>> következőt írta:
>>>
>>> Hi,
>>>
>>> I have already gone through those links. They helped me to gather the
>>> mem profile and while analyzing the data(as given in those links) I have
>>> come across the following issue:
>>>
>>> While I was running the service for 100 minutes the 'top' command output
>>> was showing Mem% as 11.1. There was no increase in mem usage since I had
>>> not called 'LoadPatternDB()' method. I have 8GB of memory on the node where
>>> I am running the service. My issue is :
>>>
>>>
>>> - Why is it showing memory accounting for around 17GB? 11.1 % of
>>> 8GB is .88GB and my node is only of 8GB. I feel the way I gathered the
>>> mem
>>> profiling was not correct ..is it ?
>>>
>>> Please let me know where am I going wrong?
>>>
>>> Thanks,
>>> Nitish
>>>
>>> On Tue, Mar 24, 2020 at 5:32 PM Nitish Saboo <nitish...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> >>There is no root analysis available in Go. Read the paper I linked
>>>> to.
>>>>
>>>> Sorry I did not get you. Which paper are you referring to?
>>>>
>>>> While I was running the service for 100 minutes the 'top' command
>>>> output was showing Mem% as 11.1. There was no increase in mem usage since I
>>>> had not called 'LoadPatternDB()' method.I have 8GB of memory on the node
>>>> where I am running the service. My issue is :
>>>>
>>>>
>>>> - Why is it showing memory accounting for around 17GB? 11.1 % of
>>>> 8GB is .88GB and my node is only of 8GB. I feel the way I gathered the
>>>> mem
>>>> profiling was not correct ..is it ?
>>>>
>>>> Please advise me what am I missing?
>>>>
>>>> Thanks,
>>>> Nitish
>>>>
>>>> On Tue, Mar 24, 2020 at 1:28 AM Robert Engels <ren...@ix.netcom.com>
>>>> wrote:
>>>>
>>>>> Yes. You have a leak in your Go code. It shows you the object types
>>>>> that are taking up all of the space. There is no root analysis available
>>>>> in
>>>>> Go. Read the paper I linked to.
>>>>>
>>>>> On Mar 23, 2020, at 9:12 AM, Nitish Saboo <nitish...@gmail.com> wrote:
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I used something like the following to generate a memprof for 100
>>>>> minutes
>>>>>
>>>>> func main() {
>>>>> flag.Parse()
>>>>> if *cpuprofile != "" {
>>>>> f, err := os.Create(*cpuprofile)
>>>>> if err != nil {
>>>>> fmt.Println("could not create CPU profile: ", err)
>>>>> }
>>>>> defer f.Close() // error handling omitted for example
>>>>> if err := pprof.StartCPUProfile(f); err != nil {
>>>>> fmt.Print("could not start CPU profile: ", err)
>>>>> }
>>>>> defer pprof.StopCPUProfile()
>>>>> }
>>>>> timeout := time.After(100 * time.Minute)
>>>>> A_chan := make(chan bool)
>>>>> B_chan := make(chan bool)
>>>>> go util.A(A_chan)
>>>>> go util.B(B_chan)
>>>>> (..Rest of the code..)
>>>>>
>>>>> for {
>>>>> select {
>>>>> case <-A_chan:
>>>>> continue
>>>>> case <-B_chan:
>>>>> continue
>>>>> case <-timeout:
>>>>> break
>>>>> }
>>>>> break
>>>>> }
>>>>>
>>>>> if *memprofile != "" {
>>>>> count = count + 1
>>>>> fmt.Println("Generating Mem Profile:")
>>>>> fmt.Print(count)
>>>>> f, err := os.Create(*memprofile)
>>>>> if err != nil {
>>>>> fmt.Println("could not create memory profile: ", err)
>>>>> }
>>>>> defer f.Close() // error handling omitted for example
>>>>> runtime.GC() // get up-to-date statistics
>>>>> if err := pprof.WriteHeapProfile(f); err != nil {
>>>>> fmt.Println("could not write memory profile: ", err)
>>>>> }
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> /Desktop/memprof:go tool pprof --alloc_space main mem3.prof
>>>>> Fetched 1 source profiles out of 2
>>>>> File: main
>>>>> Build ID: 99b8f2b91a4e037cf4a622aa32f2c1866764e7eb
>>>>> Type: alloc_space
>>>>> Time: Mar 22, 2020 at 7:02pm (IST)
>>>>> Entering interactive mode (type "help" for commands, "o" for options)
>>>>> (pprof) top
>>>>> Showing nodes accounting for 17818.11MB, 87.65% of 20329.62MB total
>>>>> <<<<<<<<<<<<<<<
>>>>> Dropped 445 nodes (cum <= 101.65MB)
>>>>> Showing top 10 nodes out of 58
>>>>> flat flat% sum% cum cum%
>>>>> 11999.09MB 59.02% 59.02% 19800.37MB 97.40%
>>>>> home/nsaboo/project/aws.Events
>>>>> 1675.69MB 8.24% 67.27% 1911.69MB 9.40%
>>>>> home/nsaboo/project/utils.Unflatten
>>>>> 627.21MB 3.09% 70.35% 1475.10MB 7.26%
>>>>> encoding/json.mapEncoder.encode
>>>>> 626.76MB 3.08% 73.43% 626.76MB 3.08%
>>>>> encoding/json.(*Decoder).refill
>>>>> 611.95MB 3.01% 76.44% 4557.69MB 22.42%
>>>>> home/nsaboo/project/lib.format
>>>>> 569.97MB 2.80% 79.25% 569.97MB 2.80% os.(*File).WriteString
>>>>> 558.95MB 2.75% 82.00% 2034.05MB 10.01% encoding/json.Marshal
>>>>> 447.51MB 2.20% 84.20% 447.51MB 2.20% reflect.copyVal
>>>>> 356.10MB 1.75% 85.95% 432.28MB 2.13% compress/flate.NewWriter
>>>>> 344.88MB 1.70% 87.65% 590.38MB 2.90% reflect.Value.MapKeys
>>>>>
>>>>> 1)Is this the correct way of getting a memory profile?
>>>>>
>>>>> 2)I ran the service for 100 minutes on a machine with 8GB memory. How
>>>>> am I seeing memory accounting for around 17GB?
>>>>>
>>>>> 3)I understand 'flat' means memory occupied within that method, but
>>>>> how come it shot up more than the available memory? Hence, asking if the
>>>>> above process is the correct way of gathering the memory profile.
>>>>>
>>>>> Thanks,
>>>>> Nitish
>>>>>
>>>>> On Fri, Mar 13, 2020 at 6:22 PM Michael Jones <michae...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> hi. get the time at the start, check the elapsed time in your
>>>>>> infinite loop, and trigger the write/exit after a minute, 10 minutes, 100
>>>>>> minutes, ...
>>>>>>
>>>>>> On Fri, Mar 13, 2020 at 5:45 AM Nitish Saboo <nitish...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> Thanks for your response.
>>>>>>>
>>>>>>> That code looks wrong. I see the end but not the start. Look here
>>>>>>> and copy carefully:
>>>>>>>
>>>>>>> >>Since I did not want cpu profiling I omitted the start of the code
>>>>>>> and just added memory profiling part.
>>>>>>>
>>>>>>> Call at end, on way out.
>>>>>>>
>>>>>>> >>Oh yes, I missed that.I have to call memory profiling code at the
>>>>>>> end on the way out.But the thing is that it runs as a service in
>>>>>>> infinite
>>>>>>> for loop.
>>>>>>>
>>>>>>> func main() {
>>>>>>> flag.Parse()
>>>>>>> if *cpuprofile != "" {
>>>>>>> f, err := os.Create(*cpuprofile)
>>>>>>> if err != nil {
>>>>>>> fmt.Println("could not create CPU profile: ", err)
>>>>>>> }
>>>>>>> defer f.Close() // error handling omitted for example
>>>>>>> if err := pprof.StartCPUProfile(f); err != nil {
>>>>>>> fmt.Print("could not start CPU profile: ", err)
>>>>>>> }
>>>>>>> defer pprof.StopCPUProfile()
>>>>>>> }
>>>>>>>
>>>>>>> A_chan := make(chan bool)
>>>>>>> B_chan := make(chan bool)
>>>>>>> go util.A(A_chan)
>>>>>>> go util.B(B_chan)
>>>>>>> (..Rest of the code..)
>>>>>>>
>>>>>>> for {
>>>>>>> select {
>>>>>>> case <-A_chan:
>>>>>>> continue
>>>>>>> case <-B_chan:
>>>>>>> continue
>>>>>>>
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> What would be the correct way to add the memprofile code changes,
>>>>>>> since it is running in an infinite for loop ?
>>>>>>>
>>>>>>> Also, as shared by others above, there are no promises about how
>>>>>>> soon the dead allocations go away, The speed gets faster and faster
>>>>>>> version
>>>>>>> to version, and is impressive indeed now, so old versions are not the
>>>>>>> best
>>>>>>> to use, ubt even so, if the allocation feels small to th GC the urgency
>>>>>>> to
>>>>>>> free it will be low. You need to loop in allocating and see if the
>>>>>>> memory
>>>>>>> grows and grows.
>>>>>>>
>>>>>>> >> Yes, got it.I will try using the latest version of Go and check
>>>>>>> the behavior.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Nitish
>>>>>>>
>>>>>>> On Fri, Mar 13, 2020 at 6:20 AM Michael Jones <michae...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> That code looks wrong. I see the end but not the start. Look here
>>>>>>>> and copy carefully:
>>>>>>>> https://golang.org/pkg/runtime/pprof/
>>>>>>>>
>>>>>>>> Call at end, on way out.
>>>>>>>>
>>>>>>>> Also, as shared by others above, there are no promises about how
>>>>>>>> soon the dead allocations go away, The speed gets faster and faster
>>>>>>>> version
>>>>>>>> to version, and is impressive indeed now, so old versions are not the
>>>>>>>> best
>>>>>>>> to use, ubt even so, if the allocation feels small to th GC the
>>>>>>>> urgency to
>>>>>>>> free it will be low. You need to loop in allocating and see if the
>>>>>>>> memory
>>>>>>>> grows and grows.
>>>>>>>>
>>>>>>>> On Thu, Mar 12, 2020 at 9:22 AM Nitish Saboo <nitish...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I have compiled my Go binary against go version 'go1.7
>>>>>>>>> linux/amd64'.
>>>>>>>>> I added the following code change in the main function to get the
>>>>>>>>> memory profiling of my service
>>>>>>>>>
>>>>>>>>> var memprofile = flag.String("memprofile", "", "write memory
>>>>>>>>> profile to `file`")
>>>>>>>>>
>>>>>>>>> func main() {
>>>>>>>>> flag.Parse()
>>>>>>>>> if *memprofile != "" {
>>>>>>>>> f, err := os.Create(*memprofile)
>>>>>>>>> if err != nil {
>>>>>>>>> fmt.Println("could not create memory profile: ", err)
>>>>>>>>> }
>>>>>>>>> defer f.Close() // error handling omitted for example
>>>>>>>>> runtime.GC() // get up-to-date statistics
>>>>>>>>> if err := pprof.WriteHeapProfile(f); err != nil {
>>>>>>>>> fmt.Println("could not write memory profile: ", err)
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> ..
>>>>>>>>> ..
>>>>>>>>> (Rest code to follow)
>>>>>>>>>
>>>>>>>>> I ran the binary with the following command:
>>>>>>>>>
>>>>>>>>> nsaboo@ubuntu:./main -memprofile=mem.prof
>>>>>>>>>
>>>>>>>>> After running the service for couple of minutes, I stopped it and
>>>>>>>>> got the file 'mem.prof'
>>>>>>>>>
>>>>>>>>> 1)mem.prof contains the following:
>>>>>>>>>
>>>>>>>>> nsaboo@ubuntu:~/Desktop/memprof$ vim mem.prof
>>>>>>>>>
>>>>>>>>> heap profile: 0: 0 [0: 0] @ heap/1048576
>>>>>>>>>
>>>>>>>>> # runtime.MemStats
>>>>>>>>> # Alloc = 761184
>>>>>>>>> # TotalAlloc = 1160960
>>>>>>>>> # Sys = 3149824
>>>>>>>>> # Lookups = 10
>>>>>>>>> # Mallocs = 8358
>>>>>>>>> # Frees = 1981
>>>>>>>>> # HeapAlloc = 761184
>>>>>>>>> # HeapSys = 1802240
>>>>>>>>> # HeapIdle = 499712
>>>>>>>>> # HeapInuse = 1302528
>>>>>>>>> # HeapReleased = 0
>>>>>>>>> # HeapObjects = 6377
>>>>>>>>> # Stack = 294912 / 294912
>>>>>>>>> # MSpan = 22560 / 32768
>>>>>>>>> # MCache = 2400 / 16384
>>>>>>>>> # BuckHashSys = 2727
>>>>>>>>> # NextGC = 4194304
>>>>>>>>> # PauseNs = [752083 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0
>>>>>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>>>> 0 0 0
>>>>>>>>> 0 0 0 0]
>>>>>>>>> # NumGC = 1
>>>>>>>>> # DebugGC = false
>>>>>>>>>
>>>>>>>>> 2)When I tried to open the file using the following command, it
>>>>>>>>> just goes into interactive mode and shows nothing
>>>>>>>>>
>>>>>>>>> a)Output from go version go1.7 linux/amd64 for mem.prof
>>>>>>>>>
>>>>>>>>> nsaboo@ubuntu:~/Desktop/memprof$ go tool pprof mem.prof
>>>>>>>>> Entering interactive mode (type "help" for commands)
>>>>>>>>> (pprof) top
>>>>>>>>> profile is empty
>>>>>>>>> (pprof)
>>>>>>>>>
>>>>>>>>> b)Output from go version go1.12.4 linux/amd64 for mem.prof
>>>>>>>>>
>>>>>>>>> nsaboo@ubuntu:~/Desktop/memprof$ go tool pprof mem.prof
>>>>>>>>> Type: space
>>>>>>>>> No samples were found with the default sample value type.
>>>>>>>>> Try "sample_index" command to analyze different sample values.
>>>>>>>>> Entering interactive mode (type "help" for commands, "o" for
>>>>>>>>> options)
>>>>>>>>> (pprof) o
>>>>>>>>> call_tree = false
>>>>>>>>> compact_labels = true
>>>>>>>>> cumulative = flat //: [cum | flat]
>>>>>>>>> divide_by = 1
>>>>>>>>> drop_negative = false
>>>>>>>>> edgefraction = 0.001
>>>>>>>>> focus = ""
>>>>>>>>> granularity = functions //: [addresses
>>>>>>>>> | filefunctions | files | functions | lines]
>>>>>>>>> hide = ""
>>>>>>>>> ignore = ""
>>>>>>>>> mean = false
>>>>>>>>> nodecount = -1 //: default
>>>>>>>>> nodefraction = 0.005
>>>>>>>>> noinlines = false
>>>>>>>>> normalize = false
>>>>>>>>> output = ""
>>>>>>>>> prune_from = ""
>>>>>>>>> relative_percentages = false
>>>>>>>>> sample_index = space //: [objects |
>>>>>>>>> space]
>>>>>>>>> show = ""
>>>>>>>>> show_from = ""
>>>>>>>>> tagfocus = ""
>>>>>>>>> taghide = ""
>>>>>>>>> tagignore = ""
>>>>>>>>> tagshow = ""
>>>>>>>>> trim = true
>>>>>>>>> trim_path = ""
>>>>>>>>> unit = minimum
>>>>>>>>> (pprof) space
>>>>>>>>> (pprof) sample_index
>>>>>>>>> (pprof) top
>>>>>>>>> Showing nodes accounting for 0, 0% of 0 total
>>>>>>>>> flat flat% sum% cum cum%
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 3)Please let me know if it is this the correct way of getting the
>>>>>>>>> memory profiling ?
>>>>>>>>>
>>>>>>>>> 4)Can we deduce something from this memory stats that points us to
>>>>>>>>> increase in memory usage?
>>>>>>>>>
>>>>>>>>> 5)I am just thinking out loud, since I am using go1.7, can that be
>>>>>>>>> the reason for the issue of increase in memory usage that might get
>>>>>>>>> fixed
>>>>>>>>> with latest go versions ?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Nitish
>>>>>>>>>
>>>>>>>>> On Tue, Mar 10, 2020 at 6:56 AM Jake Montgomery <jake...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Monday, March 9, 2020 at 1:37:00 PM UTC-4, Nitish Saboo wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Jake,
>>>>>>>>>>>
>>>>>>>>>>> The memory usage remains constant when the rest of the service
>>>>>>>>>>> is running.Only when LoadPatternDB() method is called within the
>>>>>>>>>>> service,
>>>>>>>>>>> Memory Consumption increases which actually should not happen.
>>>>>>>>>>> I am assuming if there is a memory leak while calling this
>>>>>>>>>>> method because the memory usage then becomes constant after getting
>>>>>>>>>>> increased and then further increases on next call.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Its possible that I am not fully understanding, perhaps a
>>>>>>>>>> language problem. But from what you have written above I still don't
>>>>>>>>>> see
>>>>>>>>>> that this means you definitely have a memory leak. To test for that
>>>>>>>>>> you
>>>>>>>>>> would need to *continuously *call LoadPatternDB() and monitor
>>>>>>>>>> memory for a *considerable time*. If it eventually stabilizes to
>>>>>>>>>> a constant range then there is no leak, just normal Go-GC variation.
>>>>>>>>>> If it
>>>>>>>>>> never stops climbing, and eventually consumes all the memory, then
>>>>>>>>>> it would
>>>>>>>>>> probably be a leak. Just because it goes up after one call, or a few
>>>>>>>>>> calls
>>>>>>>>>> doe not mean there is a leak.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>> Google Groups "golang-nuts" group.
>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>> send an email to golan...@googlegroups.com.
>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/f897fdb1-8968-4435-9fe9-02e167e09a36%40googlegroups.com
>>>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/f897fdb1-8968-4435-9fe9-02e167e09a36%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>> .
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "golang-nuts" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to golan...@googlegroups.com.
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq6DC98p4M4V2QCbQFTcsL1PtOWELvg8MEcMYj9EM9ui_A%40mail.gmail.com
>>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CALjMrq6DC98p4M4V2QCbQFTcsL1PtOWELvg8MEcMYj9EM9ui_A%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>>> .
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> *Michael T. jonesmichae...@gmail.com*
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> *Michael T. jonesmichae...@gmail.com*
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "golang-nuts" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to golan...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq7_d8s%3DmS5WVgV9K1m5VCBUoep2mitvX4o%3D%2BHVqf1APmQ%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/golang-nuts/CALjMrq7_d8s%3DmS5WVgV9K1m5VCBUoep2mitvX4o%3D%2BHVqf1APmQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/6455e855-3f1f-4836-ab58-2256e97f7eef%40googlegroups.com
>> <https://groups.google.com/d/msgid/golang-nuts/6455e855-3f1f-4836-ab58-2256e97f7eef%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CALjMrq7zyBTHv8YDwZqzn8Va6oN-wb7D-Cga8p_yPg3jOK950w%40mail.gmail.com.
