I was referring to the comments about the encoder keeping state. You can reset
the encoder. You may need your own framing to do so - I’m not looking at the
gob streaming encoder docs - but if it is a decent streaming encoder it should
have a reset mechanism.
That being said it is probably easier to use lightweight encoders like
protobufs - still for best efficiency for something like a game that doesn’t
need general purpose encoding - a custom encoder is probably most flexible and
efficient.
> On Dec 23, 2020, at 6:40 PM, Artur Vianna <lordhowen...@gmail.com> wrote:
>
>
> I'm confused to which part of the thread you're referring to
>
>> On Wed, 23 Dec 2020, 21:30 Robert Engels, <reng...@ix.netcom.com> wrote:
>> That is not true. Java serialization works similarly. You can hook it do
>> that you send the metadata once during connect, and then encode the data.so
>> no a new connection only needs the metadata and can decode further stream
>> messages. You may need framing resets to simplify things and reduce the
>> overhead depending on the hierarchy of “objects and references”
>>
>>>> On Dec 23, 2020, at 6:09 PM, Artur Vianna <lordhowen...@gmail.com> wrote:
>>>>
>>>
>>> Before using gob was using encoding.BinaryMarshaler, but that would mean
>>> the user of the api would need to implement a MarshalBinary for every type,
>>> which is kind of cumbersome.
>>>
>>> An option might be to let the user choose gob, BinaryMarshaler or Json etc
>>> to best fit the use case, but that takes the simplicity of only gobs away.
>>>
>>> I did try your solution to reset the client too but i'm getting
>>> inconsistent behaviour, in one server it works and in another it doesn't
>>> ("corrupted data or unknown type"). I think synching the server and client
>>> will be error prone, while also increasing the use of network.
>>>
>>> The easiest solution now is to label the package for ≤32 players and test
>>> alternative encodings that keep the API as clean as with gob. I took a look
>>> at flatbuffers but it will be cumbersome for the user to create the
>>> builders, and i really wanted the simplest possible API.
>>>
>>> Maybe i should try UDP Broadcast too and see what happens, probably chaos :D
>>>
>>>
>>>> On Wed, 23 Dec 2020, 20:36 Axel Wagner, <axel.wagner...@googlemail.com>
>>>> wrote:
>>>> No, it wouldn't. Because the encoder keeps state about which
>>>> type-information it already sent and wouldn't sent it again - causing the
>>>> client to be unable to decode. So you'd also need a new encoder on the
>>>> server. And at that point, you're back to the status quo, with one encoder
>>>> per client and the duplication of encoding effort.
>>>>
>>>> A solution would, perhaps, be if the gob API would give you a way to send
>>>> *only* the type-info (so you could, if the connection breaks, create a new
>>>> encoder, send all the type info, and *then* multicast the encoded values).
>>>> But it doesn't.
>>>>
>>>> Really, I think it's far less effort to just use a different format (and I
>>>> would maintain that even json would probably be fine) than trying to make
>>>> this work with gob :)
>>>>
>>>> On Thu, Dec 24, 2020 at 12:20 AM Matthew Zimmerman <mzimmer...@gmail.com>
>>>> wrote:
>>>>> If you would "reset" each client with a new decoder each time you make a
>>>>> new encoder, everything should work fine. Just would take some
>>>>> coordination.
>>>>>
>>>>>> On Wed, Dec 23, 2020, 6:08 PM Artur Vianna <lordhowen...@gmail.com>
>>>>>> wrote:
>>>>>> I will look into other protocols, although for now the performance is
>>>>>> not an issue in servers with less than 100 players.
>>>>>>
>>>>>> The problem with io.MultiWriter is that a player inside the group may
>>>>>> disconnect or a new player may come in. This means a new io.MultiWriter
>>>>>> must be created each time you dispatch, since the group may have changed
>>>>>> in the meantime. This would also need a new encoder and then the
>>>>>> "duplicate type received" happens.
>>>>>>
>>>>>>> On Wed, 23 Dec 2020, 19:58 'Axel Wagner' via golang-nuts,
>>>>>>> <golang-nuts@googlegroups.com> wrote:
>>>>>>> The issue with that approach is that gob keeps state about which
>>>>>>> type-information it still has to send. So if you encode to, say, a
>>>>>>> bytes.Buffer, it would encode all type-info on every message sent,
>>>>>>> which is a significant overhead.
>>>>>>> TBH, I don't understand why `io.MultiWriter` wouldn't work. It would be
>>>>>>> helpful to see the code that causes the error message OP is seeing.
>>>>>>>
>>>>>>> However, really, gob just doesn't provide a good API for this sorta
>>>>>>> thing, as mentioned. The format itself is fine, but the stateful
>>>>>>> connection means that if you don't want to write *exactly* the same
>>>>>>> data in exactly the same order to all connections (which can perform
>>>>>>> poorly and lead to operational problems with timeouts and
>>>>>>> intermittently lost connections and the like), you are going to have a
>>>>>>> bad time.
>>>>>>> You honestly would fare better with a full-fledged RPC framework such
>>>>>>> as gRPC. Or, if you don't want the overhead of its IDL, even json.
>>>>>>> Because at least the "encode once, send to each client" is trivial to
>>>>>>> solve with that.
>>>>>>>
>>>>>>> But, that's just my 2¢ :)
>>>>>>>
>>>>>>>> On Wed, Dec 23, 2020 at 11:43 PM Robert Engels <reng...@ix.netcom.com>
>>>>>>>> wrote:
>>>>>>>> Yes, that is why you need to create your own protocol. Use the gob to
>>>>>>>> encode to a buffer then send the buffer on each of the connections
>>>>>>>> using your protocol.
>>>>>>>>
>>>>>>>>>> On Dec 23, 2020, at 4:19 PM, Matthew Zimmerman
>>>>>>>>>> <mzimmer...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> My understanding is that gob streams are unique.
>>>>>>>>>
>>>>>>>>> From https://golang.org/pkg/encoding/gob/
>>>>>>>>> "A stream of gobs is self-describing. Each data item in the stream is
>>>>>>>>> preceded by a specification of its type, expressed in terms of a
>>>>>>>>> small set of predefined types."
>>>>>>>>>
>>>>>>>>> In my own rudimentary understanding/terms, it sends the struct
>>>>>>>>> definition once, then uses shorthand for it afterwards. E.g, how
>>>>>>>>> many bytes and what order. If you mix and match streams that send
>>>>>>>>> definitions in different orders, then chaos ensues.
>>>>>>>>>
>>>>>>>>> I think this is why people use other encoders in the scenario you're
>>>>>>>>> taking about. For a one to one stream gob works great, but in this
>>>>>>>>> multi scenario I don't think it does.
>>>>>>>>>
>>>>>>>>> Matt
>>>>>>>>>
>>>>>>>>>> On Wed, Dec 23, 2020, 5:07 PM Artur Vianna <lordhowen...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> If i create a bytes.Buffer and a gob.Encoder, each time i write to a
>>>>>>>>>> group of connections i get "duplicate type received" and if i try
>>>>>>>>>> and reuse the encoder, i get "corrupted data" and "unknown type".
>>>>>>>>>> It seems i can't use both net.Conn.Write and gob.Encoder.Encode in
>>>>>>>>>> the same connection, i will try always encoding to a buffer in both
>>>>>>>>>> unicast and multicast like you said and report if it works.
>>>>>>>>>>
>>>>>>>>>>> On Wed, 23 Dec 2020, 18:49 Robert Engels, <reng...@ix.netcom.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> You need to encode once to a byte array then send the byte array on
>>>>>>>>>>> each connection.
>>>>>>>>>>>
>>>>>>>>>>>>> On Dec 23, 2020, at 3:45 PM, meera <lordhowen...@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I am trying to create a package for game servers using gob. The
>>>>>>>>>>>> current approach is an application level multicasting over TCP,
>>>>>>>>>>>> having a gob encoder and decoder for each player connection, and
>>>>>>>>>>>> set up a goroutine to receive and another to dispatch for each
>>>>>>>>>>>> one. The code for the dispatcher is here. But summarized, it
>>>>>>>>>>>> simply receives data from a channel and encodes it.
>>>>>>>>>>>>
>>>>>>>>>>>> The problem is that if i want to transmit a single piece of data
>>>>>>>>>>>> to all players, this piece of data is encoded again and again for
>>>>>>>>>>>> each connection, doing duplicate work. With less than 100 players
>>>>>>>>>>>> this is not a problem, but with 300+ my machine is at almost 100%
>>>>>>>>>>>> usage and the profiler shows that most of it is spent on encoding.
>>>>>>>>>>>> Here's the issue on github.
>>>>>>>>>>>>
>>>>>>>>>>>> I tryied using a io.MultiWriter but gob complains of duplicate
>>>>>>>>>>>> type received, and if i try to write the raw bytes from the
>>>>>>>>>>>> gob.Encoder i get corrupted data. An option is using UDP
>>>>>>>>>>>> Broadcasting but since gob expects a stream, i'm affraid i will
>>>>>>>>>>>> run into unexpected behavior when packets start to be lost and
>>>>>>>>>>>> fragmented.
>>>>>>>>>>>>
>>>>>>>>>>>> Does gob expect a single encoder and decoder to own the stream?
>>>>>>>>>>>> Not allowing two encoders on the server for one decoder on the
>>>>>>>>>>>> client?
>>>>>>>>>>>> --
>>>>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>>>>> Groups "golang-nuts" group.
>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/0562184e-bbcc-44c9-adbf-37e8d5411c7cn%40googlegroups.com.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>>> Groups "golang-nuts" group.
>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBXN46idvqUbCsGs%2BZbZt%2BCj4MowJ4Ozj3_U9_6-68OWDw%40mail.gmail.com.
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "golang-nuts" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/214752B6-2666-4892-A9B8-E4BC4127FD42%40ix.netcom.com.
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "golang-nuts" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGWtULh8Q3Jqu_gq5m5Si4PvJ1oVSZY7DVhu%3D6hGK83bg%40mail.gmail.com.
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "golang-nuts" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBUsmp2sbiEh%3D3z0cC9EhjLig%2B8exXyA05YngBJ-tsC_uA%40mail.gmail.com.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBU%3DZBR8O9na44vsXCg9NXrH9tdo5xxuBjYqMOYoGU4N0w%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/9D99C078-8DFA-4C30-8258-46F10709FAAF%40ix.netcom.com.
