I will definitely explore your ideas, thanks for the insights. I'll let you
know how it turned out in the end :D
Thanks
On Thu, 24 Dec 2020, 06:59 Axel Wagner, <axel.wagner...@googlemail.com>
wrote:
> On Thu, Dec 24, 2020 at 6:35 AM Artur Vianna <lordhowen...@gmail.com>
> wrote:
>
>> Exposing the bytes would hurt the abstraction.
>>
>
> I strongly disagree with this. Exposing the bytes would improve the
> abstraction, by making the transmission- and encoding method orthogonal.
> And as I said, you can still provide just as convenient a wrapper around it.
>
>
>> I will certainly look into Json and maybe MessagePack. But first i may
>> fiddle with the gob source code, if my sanity allows it. If i can control
>> when the type information is sent i may be able to fix this without
>> changing the API. The ideal enconding would be a stateless gob, with the
>> proper decoding/encoding "machines" set on either side when you call
>> gob.Register, but i'm not sure of the feasibility of that.
>>
>
> Sure, that's definitely possible. I factored that in, when I said "I think
> it's far less effort to just use a different format than trying to make
> this work with gob".
>
> But, of course, you do you. :) The decision of what API to expose and what
> encoding scheme to use and how much work you are putting into what piece is
> ultimately up to you :)
>
>
>>
>> On Wed, 23 Dec 2020, 21:55 Axel Wagner, <axel.wagner...@googlemail.com>
>> wrote:
>>
>>> On Thu, Dec 24, 2020 at 1:08 AM Artur Vianna <lordhowen...@gmail.com>
>>> wrote:
>>>
>>>> Before using gob was using encoding.BinaryMarshaler, but that would
>>>> mean the user of the api would need to implement a MarshalBinary for every
>>>> type, which is kind of cumbersome.
>>>>
>>>
>>>> An option might be to let the user choose gob, BinaryMarshaler or Json
>>>> etc to best fit the use case, but that takes the simplicity of only gobs
>>>> away.
>>>>
>>>
>>> I am all in favor of API simplicity, but gob just isn't super useful for
>>> this. Not to repeat myself, but you should really at least try JSON - it
>>> provides exactly the same convenience as gob, but doesn't suffer these
>>> problems. It might have a bit more overhead and might even be costlier to
>>> encode - but the savings from being able to eliminate duplicate effort
>>> should offset that (and I'm not even super convinced - I don't think gob is
>>> the most well-optimized encoding in the stdlib).
>>>
>>> But IMO, if you provide an API, the best solution is to a) just use
>>> `[]byte` at the base-layer and then b) provide convenience-wrappers around
>>> that for other formats. This lets the users decide what they want (they
>>> might want something completely different anyway) while still providing a
>>> decently convenient API for simple uses.
>>>
>>>
>>>>
>>>> I did try your solution to reset the client too but i'm getting
>>>> inconsistent behaviour, in one server it works and in another it doesn't
>>>> ("corrupted data or unknown type"). I think synching the server and client
>>>> will be error prone, while also increasing the use of network.
>>>>
>>>> The easiest solution now is to label the package for ≤32 players and
>>>> test alternative encodings that keep the API as clean as with gob. I took a
>>>> look at flatbuffers but it will be cumbersome for the user to create the
>>>> builders, and i really wanted the simplest possible API.
>>>>
>>>> Maybe i should try UDP Broadcast too and see what happens, probably
>>>> chaos :D
>>>>
>>>>
>>>> On Wed, 23 Dec 2020, 20:36 Axel Wagner, <axel.wagner...@googlemail.com>
>>>> wrote:
>>>>
>>>>> No, it wouldn't. Because the encoder keeps state about which
>>>>> type-information it already sent and wouldn't sent it again - causing the
>>>>> client to be unable to decode. So you'd also need a new encoder on the
>>>>> server. And at that point, you're back to the status quo, with one encoder
>>>>> per client and the duplication of encoding effort.
>>>>>
>>>>> A solution would, perhaps, be if the gob API would give you a way to
>>>>> send *only* the type-info (so you could, if the connection breaks, create
>>>>> a
>>>>> new encoder, send all the type info, and *then* multicast the encoded
>>>>> values). But it doesn't.
>>>>>
>>>>> Really, I think it's far less effort to just use a different format
>>>>> (and I would maintain that even json would probably be fine) than trying
>>>>> to
>>>>> make this work with gob :)
>>>>>
>>>>> On Thu, Dec 24, 2020 at 12:20 AM Matthew Zimmerman <
>>>>> mzimmer...@gmail.com> wrote:
>>>>>
>>>>>> If you would "reset" each client with a new decoder each time you
>>>>>> make a new encoder, everything should work fine. Just would take some
>>>>>> coordination.
>>>>>>
>>>>>> On Wed, Dec 23, 2020, 6:08 PM Artur Vianna <lordhowen...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I will look into other protocols, although for now the performance
>>>>>>> is not an issue in servers with less than 100 players.
>>>>>>>
>>>>>>> The problem with io.MultiWriter is that a player inside the group
>>>>>>> may disconnect or a new player may come in. This means a new
>>>>>>> io.MultiWriter
>>>>>>> must be created each time you dispatch, since the group may have
>>>>>>> changed in
>>>>>>> the meantime. This would also need a new encoder and then the "duplicate
>>>>>>> type received" happens.
>>>>>>>
>>>>>>> On Wed, 23 Dec 2020, 19:58 'Axel Wagner' via golang-nuts, <
>>>>>>> golang-nuts@googlegroups.com> wrote:
>>>>>>>
>>>>>>>> The issue with that approach is that gob keeps state about which
>>>>>>>> type-information it still has to send. So if you encode to, say, a
>>>>>>>> bytes.Buffer, it would encode all type-info on every message sent,
>>>>>>>> which is
>>>>>>>> a significant overhead.
>>>>>>>> TBH, I don't understand why `io.MultiWriter` wouldn't work. It
>>>>>>>> would be helpful to see the code that causes the error message OP is
>>>>>>>> seeing.
>>>>>>>>
>>>>>>>> However, really, gob just doesn't provide a good API for this sorta
>>>>>>>> thing, as mentioned. The format itself is fine, but the stateful
>>>>>>>> connection
>>>>>>>> means that if you don't want to write *exactly* the same data in
>>>>>>>> exactly
>>>>>>>> the same order to all connections (which can perform poorly and lead to
>>>>>>>> operational problems with timeouts and intermittently lost connections
>>>>>>>> and
>>>>>>>> the like), you are going to have a bad time.
>>>>>>>> You honestly would fare better with a full-fledged RPC framework
>>>>>>>> such as gRPC. Or, if you don't want the overhead of its IDL, even json.
>>>>>>>> Because at least the "encode once, send to each client" is trivial to
>>>>>>>> solve
>>>>>>>> with that.
>>>>>>>>
>>>>>>>> But, that's just my 2¢ :)
>>>>>>>>
>>>>>>>> On Wed, Dec 23, 2020 at 11:43 PM Robert Engels <
>>>>>>>> reng...@ix.netcom.com> wrote:
>>>>>>>>
>>>>>>>>> Yes, that is why you need to create your own protocol. Use the gob
>>>>>>>>> to encode to a buffer then send the buffer on each of the connections
>>>>>>>>> using
>>>>>>>>> your protocol.
>>>>>>>>>
>>>>>>>>> On Dec 23, 2020, at 4:19 PM, Matthew Zimmerman <
>>>>>>>>> mzimmer...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> My understanding is that gob streams are unique.
>>>>>>>>>
>>>>>>>>> From https://golang.org/pkg/encoding/gob/
>>>>>>>>> "A stream of gobs is self-describing. Each data item in the stream
>>>>>>>>> is preceded by a specification of its type, expressed in terms of a
>>>>>>>>> small
>>>>>>>>> set of predefined types."
>>>>>>>>>
>>>>>>>>> In my own rudimentary understanding/terms, it sends the struct
>>>>>>>>> definition once, then uses shorthand for it afterwards. E.g, how many
>>>>>>>>> bytes and what order. If you mix and match streams that send
>>>>>>>>> definitions
>>>>>>>>> in different orders, then chaos ensues.
>>>>>>>>>
>>>>>>>>> I think this is why people use other encoders in the scenario
>>>>>>>>> you're taking about. For a one to one stream gob works great, but in
>>>>>>>>> this
>>>>>>>>> multi scenario I don't think it does.
>>>>>>>>>
>>>>>>>>> Matt
>>>>>>>>>
>>>>>>>>> On Wed, Dec 23, 2020, 5:07 PM Artur Vianna <lordhowen...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> If i create a bytes.Buffer and a gob.Encoder, each time i write
>>>>>>>>>> to a group of connections i get "duplicate type received" and if i
>>>>>>>>>> try and
>>>>>>>>>> reuse the encoder, i get "corrupted data" and "unknown type".
>>>>>>>>>> It seems i can't use both net.Conn.Write and gob.Encoder.Encode
>>>>>>>>>> in the same connection, i will try always encoding to a buffer in
>>>>>>>>>> both
>>>>>>>>>> unicast and multicast like you said and report if it works.
>>>>>>>>>>
>>>>>>>>>> On Wed, 23 Dec 2020, 18:49 Robert Engels, <reng...@ix.netcom.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> You need to encode once to a byte array then send the byte array
>>>>>>>>>>> on each connection.
>>>>>>>>>>>
>>>>>>>>>>> On Dec 23, 2020, at 3:45 PM, meera <lordhowen...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I am trying to create a package for game servers using gob. The
>>>>>>>>>>> current approach is an application level multicasting over TCP,
>>>>>>>>>>> having a
>>>>>>>>>>> gob encoder and decoder for each player connection, and set up a
>>>>>>>>>>> goroutine
>>>>>>>>>>> to receive and another to dispatch for each one. The code for the
>>>>>>>>>>> dispatcher is here. But summarized, it simply receives data from a
>>>>>>>>>>> channel
>>>>>>>>>>> and encodes it.
>>>>>>>>>>>
>>>>>>>>>>> The problem is that if i want to transmit a single piece of data
>>>>>>>>>>> to all players, this piece of data is encoded again and again for
>>>>>>>>>>> each
>>>>>>>>>>> connection, doing duplicate work. With less than 100 players this
>>>>>>>>>>> is not a
>>>>>>>>>>> problem, but with 300+ my machine is at almost 100% usage and the
>>>>>>>>>>> profiler
>>>>>>>>>>> shows that most of it is spent on encoding. Here's the issue on
>>>>>>>>>>> github.
>>>>>>>>>>>
>>>>>>>>>>> I tryied using a io.MultiWriter but gob complains of duplicate
>>>>>>>>>>> type received, and if i try to write the raw bytes from the
>>>>>>>>>>> gob.Encoder i
>>>>>>>>>>> get corrupted data. An option is using UDP Broadcasting but since
>>>>>>>>>>> gob
>>>>>>>>>>> expects a stream, i'm affraid i will run into unexpected behavior
>>>>>>>>>>> when
>>>>>>>>>>> packets start to be lost and fragmented.
>>>>>>>>>>>
>>>>>>>>>>> Does gob expect a single encoder and decoder to own the stream?
>>>>>>>>>>> Not allowing two encoders on the server for one decoder on the
>>>>>>>>>>> client?
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "golang-nuts" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>> it, send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/0562184e-bbcc-44c9-adbf-37e8d5411c7cn%40googlegroups.com
>>>>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/0562184e-bbcc-44c9-adbf-37e8d5411c7cn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>> Google Groups "golang-nuts" group.
>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBXN46idvqUbCsGs%2BZbZt%2BCj4MowJ4Ozj3_U9_6-68OWDw%40mail.gmail.com
>>>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBXN46idvqUbCsGs%2BZbZt%2BCj4MowJ4Ozj3_U9_6-68OWDw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>>>> .
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "golang-nuts" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/214752B6-2666-4892-A9B8-E4BC4127FD42%40ix.netcom.com
>>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/214752B6-2666-4892-A9B8-E4BC4127FD42%40ix.netcom.com?utm_medium=email&utm_source=footer>
>>>>>>>>> .
>>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "golang-nuts" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGWtULh8Q3Jqu_gq5m5Si4PvJ1oVSZY7DVhu%3D6hGK83bg%40mail.gmail.com
>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGWtULh8Q3Jqu_gq5m5Si4PvJ1oVSZY7DVhu%3D6hGK83bg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "golang-nuts" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBUsmp2sbiEh%3D3z0cC9EhjLig%2B8exXyA05YngBJ-tsC_uA%40mail.gmail.com
>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBUsmp2sbiEh%3D3z0cC9EhjLig%2B8exXyA05YngBJ-tsC_uA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBWmiQbVBxQ2FcWguzbd7Y62LUjmQ%3DY0qKe%3D4M0WKgoubg%40mail.gmail.com.
