Exposing the bytes would hurt the abstraction. Game servers usually only
process data in certain intervals (say 20 times per second), that means i
would either need to expose the net.Conn for reading, or creating a buffer
for each connection and acumulate the data on memory, so that each time the
server updates, the data is available (without filling up the TCP buffer).
Also this data is processed into a single place, where you change the state
of the server.
Players are expected to connect any time the server is up, or none at all
(and the server might still update it's state without the players), that's
why i needed to create a new io.MultiWriter every Dispatch.
I will certainly look into Json and maybe MessagePack. But first i may
fiddle with the gob source code, if my sanity allows it. If i can control
when the type information is sent i may be able to fix this without
changing the API. The ideal enconding would be a stateless gob, with the
proper decoding/encoding "machines" set on either side when you call
gob.Register, but i'm not sure of the feasibility of that.
On Wed, 23 Dec 2020, 21:55 Axel Wagner, <axel.wagner...@googlemail.com>
wrote:
> On Thu, Dec 24, 2020 at 1:08 AM Artur Vianna <lordhowen...@gmail.com>
> wrote:
>
>> Before using gob was using encoding.BinaryMarshaler, but that would mean
>> the user of the api would need to implement a MarshalBinary for every type,
>> which is kind of cumbersome.
>>
>
>> An option might be to let the user choose gob, BinaryMarshaler or Json
>> etc to best fit the use case, but that takes the simplicity of only gobs
>> away.
>>
>
> I am all in favor of API simplicity, but gob just isn't super useful for
> this. Not to repeat myself, but you should really at least try JSON - it
> provides exactly the same convenience as gob, but doesn't suffer these
> problems. It might have a bit more overhead and might even be costlier to
> encode - but the savings from being able to eliminate duplicate effort
> should offset that (and I'm not even super convinced - I don't think gob is
> the most well-optimized encoding in the stdlib).
>
> But IMO, if you provide an API, the best solution is to a) just use
> `[]byte` at the base-layer and then b) provide convenience-wrappers around
> that for other formats. This lets the users decide what they want (they
> might want something completely different anyway) while still providing a
> decently convenient API for simple uses.
>
>
>>
>> I did try your solution to reset the client too but i'm getting
>> inconsistent behaviour, in one server it works and in another it doesn't
>> ("corrupted data or unknown type"). I think synching the server and client
>> will be error prone, while also increasing the use of network.
>>
>> The easiest solution now is to label the package for ≤32 players and test
>> alternative encodings that keep the API as clean as with gob. I took a look
>> at flatbuffers but it will be cumbersome for the user to create the
>> builders, and i really wanted the simplest possible API.
>>
>> Maybe i should try UDP Broadcast too and see what happens, probably chaos
>> :D
>>
>>
>> On Wed, 23 Dec 2020, 20:36 Axel Wagner, <axel.wagner...@googlemail.com>
>> wrote:
>>
>>> No, it wouldn't. Because the encoder keeps state about which
>>> type-information it already sent and wouldn't sent it again - causing the
>>> client to be unable to decode. So you'd also need a new encoder on the
>>> server. And at that point, you're back to the status quo, with one encoder
>>> per client and the duplication of encoding effort.
>>>
>>> A solution would, perhaps, be if the gob API would give you a way to
>>> send *only* the type-info (so you could, if the connection breaks, create a
>>> new encoder, send all the type info, and *then* multicast the encoded
>>> values). But it doesn't.
>>>
>>> Really, I think it's far less effort to just use a different format (and
>>> I would maintain that even json would probably be fine) than trying to make
>>> this work with gob :)
>>>
>>> On Thu, Dec 24, 2020 at 12:20 AM Matthew Zimmerman <mzimmer...@gmail.com>
>>> wrote:
>>>
>>>> If you would "reset" each client with a new decoder each time you make
>>>> a new encoder, everything should work fine. Just would take some
>>>> coordination.
>>>>
>>>> On Wed, Dec 23, 2020, 6:08 PM Artur Vianna <lordhowen...@gmail.com>
>>>> wrote:
>>>>
>>>>> I will look into other protocols, although for now the performance is
>>>>> not an issue in servers with less than 100 players.
>>>>>
>>>>> The problem with io.MultiWriter is that a player inside the group may
>>>>> disconnect or a new player may come in. This means a new io.MultiWriter
>>>>> must be created each time you dispatch, since the group may have changed
>>>>> in
>>>>> the meantime. This would also need a new encoder and then the "duplicate
>>>>> type received" happens.
>>>>>
>>>>> On Wed, 23 Dec 2020, 19:58 'Axel Wagner' via golang-nuts, <
>>>>> golang-nuts@googlegroups.com> wrote:
>>>>>
>>>>>> The issue with that approach is that gob keeps state about which
>>>>>> type-information it still has to send. So if you encode to, say, a
>>>>>> bytes.Buffer, it would encode all type-info on every message sent, which
>>>>>> is
>>>>>> a significant overhead.
>>>>>> TBH, I don't understand why `io.MultiWriter` wouldn't work. It would
>>>>>> be helpful to see the code that causes the error message OP is seeing.
>>>>>>
>>>>>> However, really, gob just doesn't provide a good API for this sorta
>>>>>> thing, as mentioned. The format itself is fine, but the stateful
>>>>>> connection
>>>>>> means that if you don't want to write *exactly* the same data in exactly
>>>>>> the same order to all connections (which can perform poorly and lead to
>>>>>> operational problems with timeouts and intermittently lost connections
>>>>>> and
>>>>>> the like), you are going to have a bad time.
>>>>>> You honestly would fare better with a full-fledged RPC framework such
>>>>>> as gRPC. Or, if you don't want the overhead of its IDL, even json.
>>>>>> Because
>>>>>> at least the "encode once, send to each client" is trivial to solve with
>>>>>> that.
>>>>>>
>>>>>> But, that's just my 2¢ :)
>>>>>>
>>>>>> On Wed, Dec 23, 2020 at 11:43 PM Robert Engels <reng...@ix.netcom.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, that is why you need to create your own protocol. Use the gob
>>>>>>> to encode to a buffer then send the buffer on each of the connections
>>>>>>> using
>>>>>>> your protocol.
>>>>>>>
>>>>>>> On Dec 23, 2020, at 4:19 PM, Matthew Zimmerman <mzimmer...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> My understanding is that gob streams are unique.
>>>>>>>
>>>>>>> From https://golang.org/pkg/encoding/gob/
>>>>>>> "A stream of gobs is self-describing. Each data item in the stream
>>>>>>> is preceded by a specification of its type, expressed in terms of a
>>>>>>> small
>>>>>>> set of predefined types."
>>>>>>>
>>>>>>> In my own rudimentary understanding/terms, it sends the struct
>>>>>>> definition once, then uses shorthand for it afterwards. E.g, how many
>>>>>>> bytes and what order. If you mix and match streams that send
>>>>>>> definitions
>>>>>>> in different orders, then chaos ensues.
>>>>>>>
>>>>>>> I think this is why people use other encoders in the scenario you're
>>>>>>> taking about. For a one to one stream gob works great, but in this
>>>>>>> multi
>>>>>>> scenario I don't think it does.
>>>>>>>
>>>>>>> Matt
>>>>>>>
>>>>>>> On Wed, Dec 23, 2020, 5:07 PM Artur Vianna <lordhowen...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> If i create a bytes.Buffer and a gob.Encoder, each time i write to
>>>>>>>> a group of connections i get "duplicate type received" and if i try and
>>>>>>>> reuse the encoder, i get "corrupted data" and "unknown type".
>>>>>>>> It seems i can't use both net.Conn.Write and gob.Encoder.Encode in
>>>>>>>> the same connection, i will try always encoding to a buffer in both
>>>>>>>> unicast
>>>>>>>> and multicast like you said and report if it works.
>>>>>>>>
>>>>>>>> On Wed, 23 Dec 2020, 18:49 Robert Engels, <reng...@ix.netcom.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> You need to encode once to a byte array then send the byte array
>>>>>>>>> on each connection.
>>>>>>>>>
>>>>>>>>> On Dec 23, 2020, at 3:45 PM, meera <lordhowen...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I am trying to create a package for game servers using gob. The
>>>>>>>>> current approach is an application level multicasting over TCP,
>>>>>>>>> having a
>>>>>>>>> gob encoder and decoder for each player connection, and set up a
>>>>>>>>> goroutine
>>>>>>>>> to receive and another to dispatch for each one. The code for the
>>>>>>>>> dispatcher is here. But summarized, it simply receives data from a
>>>>>>>>> channel
>>>>>>>>> and encodes it.
>>>>>>>>>
>>>>>>>>> The problem is that if i want to transmit a single piece of data
>>>>>>>>> to all players, this piece of data is encoded again and again for each
>>>>>>>>> connection, doing duplicate work. With less than 100 players this is
>>>>>>>>> not a
>>>>>>>>> problem, but with 300+ my machine is at almost 100% usage and the
>>>>>>>>> profiler
>>>>>>>>> shows that most of it is spent on encoding. Here's the issue on
>>>>>>>>> github.
>>>>>>>>>
>>>>>>>>> I tryied using a io.MultiWriter but gob complains of duplicate
>>>>>>>>> type received, and if i try to write the raw bytes from the
>>>>>>>>> gob.Encoder i
>>>>>>>>> get corrupted data. An option is using UDP Broadcasting but since gob
>>>>>>>>> expects a stream, i'm affraid i will run into unexpected behavior when
>>>>>>>>> packets start to be lost and fragmented.
>>>>>>>>>
>>>>>>>>> Does gob expect a single encoder and decoder to own the stream?
>>>>>>>>> Not allowing two encoders on the server for one decoder on the client?
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "golang-nuts" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/0562184e-bbcc-44c9-adbf-37e8d5411c7cn%40googlegroups.com
>>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/0562184e-bbcc-44c9-adbf-37e8d5411c7cn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>> .
>>>>>>>>>
>>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "golang-nuts" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBXN46idvqUbCsGs%2BZbZt%2BCj4MowJ4Ozj3_U9_6-68OWDw%40mail.gmail.com
>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBXN46idvqUbCsGs%2BZbZt%2BCj4MowJ4Ozj3_U9_6-68OWDw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "golang-nuts" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/golang-nuts/214752B6-2666-4892-A9B8-E4BC4127FD42%40ix.netcom.com
>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/214752B6-2666-4892-A9B8-E4BC4127FD42%40ix.netcom.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "golang-nuts" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGWtULh8Q3Jqu_gq5m5Si4PvJ1oVSZY7DVhu%3D6hGK83bg%40mail.gmail.com
>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGWtULh8Q3Jqu_gq5m5Si4PvJ1oVSZY7DVhu%3D6hGK83bg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "golang-nuts" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBUsmp2sbiEh%3D3z0cC9EhjLig%2B8exXyA05YngBJ-tsC_uA%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBUsmp2sbiEh%3D3z0cC9EhjLig%2B8exXyA05YngBJ-tsC_uA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBUU%3D1iLWwjKiDTVaeRKL1vVh5LtrQJW_ojWaONb6Pj2Mg%40mail.gmail.com.
