Sure, feel free to do what you are comfortable with. I just couldn't
understand why there is a need for the Update function at all.
Hope that your Go transition goes smoothly :)
-Agniva
On Thu, 27 Sep 2018 at 21:29 Michael Ellis <michael.f.el...@gmail.com>
wrote:
> Thanks for the feedback, Agni. I think I may not have included enough
> detail in my original post.
>
> At present, the app is in the middle of transitioning to an all-Go
> implementation. For business reasons, that process will be stretched out
> over the next several months or longer. In the meantime, it needs to keep
> working so I've already implemented something quite similar to what you
> describe by porting the busiest Python processes first and providing a
> socket interface that replaces the ZeroMQ interface in a way that allows
> the calls from the remaining Python processes to have the same signature
> and behavior as before. That's working quite well so far and we're already
> seeing better performance and improved reliability.
>
> I should also mention that the target for the system is an inexpensive
> Linux SBC, a Raspberry Pi. In the pure Python version, the overhead for
> serializing, transmitting and de-serializing data had grown to use far more
> CPU than seemed wise in a system that needs to run reliably for months at a
> time at customer sites around the world.
>
> Within the collection of processes that are now goroutines, I can't see a
> benefit to incurring the overhead to marshal and unmarshal the data if
> there's a concurrency-safe way to perform updates and get snapshots of the
> state data.
>
> Sending functions over a channel seems to be a match made in heaven for
> our use cases. It nicely supports our application logic that parcels out
> responsibility for updating pieces of the state struct to long-running
> goroutines. Most of them are loops that fetch a snapshot of the state,
> inspect it to make decisions about what actions to take, perform the
> actions and update the state.
>
> In that context, sending anonymous update functions wrapped in struct with
> a disposable "done" channel doesn't seem overcomplicated at all. The
> update actions in my first post boil down to three lines that can be
> wrapped up in a single function, e.g.
>
> func Update(f func(*Big)) {
> u := update{make(chan struct{}), f}
> upch <- u // upch is a toplevel var visible to all goroutines
> <-u.done
> }
>
> and an (oversimplified) goroutine that, say, reads a new value from a
> sensor looks like
>
> func gopher() {
> var newA int
> f := func(b *Big) {
> b.A = newA
> }
> for {
> newA = ReadSensor()
> Update(f)
> }
> }
>
> As I see it, the beauties of this approach are:
>
> 1. f is locally defined and therefore has access to gopher's locals,
> 2. gopher has no direct access to the instance of Big that holds the
> state.
> 3. Update pauses gopher until the state update has completed in the
> single goroutine that reads from upch. So no worries about gopher changing
> its own locals before the update is complete.
> 4. Benchmarking shows it's more than fast enough for our needs.
> 5. The compiler catches any type mismatches instead of depending on
> the runtime to report it at deserialization.
>
>
> Sorry for the long-winded reply. I really appreciate the feedback and the
> opportunity to learn from you and others who've been using Go far longer
> than I.
>
>
>
>
>
>
>
> On Thu, Sep 27, 2018 at 12:48 AM Agniva De Sarker <
> agniva.quicksil...@gmail.com> wrote:
>
>> I think you are overcomplicating this a bit. It seems like a simple
>> pattern of broadcasting a change to multiple agents. You send the change
>> over a REQ-REP pair, and broadcast it to others over a PUB-SUB pair.
>>
>> Why do you need to copy the struct again ? Just get the struct from the
>> REP socket and push it to the PUB socket ?
>>
>> Here is what I would do -
>>
>> Use protobuf to marshal/unmarshal the config struct when
>> sending/receiving it through the socket. That way you have perfect interop
>> between Go and Python.
>>
>> And use a for-select pattern to do the job of receiving and broadcasting.
>> Pseudo code as follows
>>
>> package main
>>
>> func main() {
>> // setup REP socket in a goroutine, unmarshal the msg, send the struct to
>> updateChan
>> // create PUB socket, store it in mem somewhere
>>
>> go run()
>>
>> // setup signal handlers, and send signal to done chan when
>> ctrl-c is received.
>> }
>>
>> func run() {
>> for {
>> select {
>> case msg := <-updateChan:
>> // push msg to the PUB socket
>> case <-done:
>> // you have to send signal to this from main()
>> return
>> }
>> }
>> }
>>
>> Isn't this what you are trying to do ?
>>
>>
>>
>> On Wednesday, 26 September 2018 00:09:07 UTC+5:30, Michael Ellis wrote:
>>>
>>> Hi, new gopher here.
>>> I considered asking this on SO, but they (rightly, IMO) discourage "Is
>>> this a good way to do it?" questions. Hope that's ok here.
>>>
>>> By way of background, I'm porting a largish industrial control
>>> application from Python to Go. The Python version uses multiple processes
>>> (about a dozen in all) communicating over ZeroMQ. One process, called the
>>> statehouse, controls access to the application state. The others obtain
>>> copies and send updates over REQ sockets. The data are serialized as JSON
>>> objects that map nicely to Python dicts.
>>>
>>> Since there's no direct equivalent in Go to a Python dict that can hold
>>> a mixture of arbitrary types, I need to use a struct to represent the
>>> state. No problem with that but I've been struggling with how to allow the
>>> goroutines that will replace the Python processes to read and write to the
>>> state struct with concurrency safety.
>>>
>>> This morning I came up with an idea to send functions over a channel to
>>> the main routine. I put together a little test program and after some
>>> refinements it looks promising. Some rough benchmarking shows I can get a
>>> million updates in under 1 second on a 2012 vintage Mac Mini. That's more
>>> than good enough for this application where the time between events is
>>> usually more than 100 milliseconds.
>>>
>>> Here's the link to my test on the Go Playground:
>>> https://play.golang.org/p/8iWvwnqBNYl . It runs there except that the
>>> elapsed time comes back 0 and the prints from the second goroutine don't
>>> show up. I think that's got something to do with the artificial clock in
>>> the playground. It works fine when I run it locally. I've pasted the code
>>> at the bottom of this message.
>>>
>>> So my big questions are:
>>>
>>> - Is this actually concurrency safe as long as all goroutines only
>>> use the update mechanism to read and write?
>>> - Is there a more idiomatic way to do it that performs as well or
>>> better?
>>> - What are the potential problems if this is scaled to a couple
>>> dozen goroutines?
>>> - Does it sacrifice clarity for cleverness? (not that it's all that
>>> clever, mind you, but I need to think about handing this off to my
>>> client's
>>> staff.)
>>>
>>>
>>> Thanks very much,
>>> Mike Ellis
>>>
>>> code follows ...
>>>
>>> package main
>>>
>>> import (
>>> "fmt"
>>> "time"
>>> )
>>>
>>>
>>> // Big defines the application's state variables
>>> type Big struct {
>>> A int
>>> B string
>>> /* and hundreds more */
>>> }
>>>
>>>
>>> // update is a struct that contains a function that updates a Big and
>>> // a signal channel to be closed when the update is complete. An update
>>> // may also be used to obtain a current copy of a Big by coding f to
>>> // do so. (See gopher2 below.)
>>> type update struct {
>>> done chan struct{}
>>> f func(*Big)
>>> }
>>>
>>>
>>> // upch is a channel from which main receives updates.
>>> var upch = make(chan update)
>>>
>>>
>>> // gopher defines a function that updates a member of a Big and
>>> // sends updates via upch. After each send it waits for main to
>>> // close the update's done channel.
>>> func gopher() {
>>> var newA int
>>> f := func(b *Big) {
>>> b.A = newA
>>> }
>>> for i := 0; i < n; i++ {
>>> newA = i
>>> u := update{make(chan struct{}), f}
>>> upch <- u
>>> <-u.done
>>> }
>>> }
>>>
>>>
>>> // gopher2 uses an update struct to obtain a current copy of a Big
>>> // every 100 microseconds.
>>> func gopher2() {
>>> var copied Big
>>> f := func(b *Big) {
>>> copied = *b
>>> }
>>> for {
>>> time.Sleep(100 * time.Microsecond)
>>> u := update{make(chan struct{}), f}
>>> upch <- u
>>> <-u.done
>>> fmt.Println(copied)
>>> }
>>> }
>>>
>>>
>>> // main creates a Big, launches gopher and waits on the update channel.
>>> When
>>> // an update, u, arrives it runs u.f and then closes u.done.
>>> func main() {
>>> var state = Big{-1, "foo"}
>>> fmt.Println(state) // --> {-1, "foo"}
>>> go gopher()
>>> go gopher2()
>>> start := time.Now()
>>> for i := 0; i < n; i++ {
>>> u := <-upch
>>> u.f(&state)
>>> close(u.done)
>>> }
>>> perUpdate := time.Since(start).Nanoseconds() / int64(n) // Note:
>>> always 0 in playground
>>> fmt.Printf("%d updates, %d ns per update.\n", n, perUpdate)
>>> fmt.Println(state) // --> {n-1, "foo"}
>>> }
>>>
>>>
>>> var n = 1000 // number of updates to send and receive
>>>
>>>
>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "golang-nuts" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/golang-nuts/KiNpAxobbec/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> golang-nuts+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> Write music faster with Tbon.
> <https://github.com/Michael-F-Ellis/tbon/#tbon>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
