> On Nov 22, 2015, at 17:51, Oon-Ee Ng <ngoonee.t...@gmail.com> wrote: > > I've just (to my surprise) hit this. As I understand from searching > around, AMP messages are limited to ~64k due to the length prefix > being 16-bit. A change in my internal data being sent (using dicts > rather than lists) kicked one of my messages to way over that limit.
I'm sorry that this was an unpleasant surprise. I wish that we had a better way of getting this across up-front :-). However, it seems like the length limit is doing its job in terms of constraining your protocol design to not have individual messages "hog" the wire... > There's a bit of discussion here - > http://twistedmatrix.com/pipermail/twisted-python/2014-November/028947.html > > Is there an internal twisted solution planned, or should I go ahead > and roll my own paging code? If the latter (as I strongly suspect), > could I get some comments on this idea:- Definitely the latter if you have a short time frame. How big are your messages? If your limit is still fairly small (5M, let's say) but much bigger than 64k there are other options you can use. > Original amp.Command had a single argument (amp.ListOf(amp.String()) > and no response > Modified amp.Command, 4 arguments and 1 response > ID (sequentially generated by producer) - amp.Integer() > CurPage - amp.Integer() > TotalPage - amp.Integer() > ActualData - amp.ListOf(amp.String()) > Response - RecievedPage - amp.Integer() Implementing a paging API like this is exactly what the length limit is designed to encourage you to do - it is much more flexible, since you can request a subset of pages, and continue receiving things other than pages while the data is being streamed to you. > Questions:- > 1. ID is so the client can be sure not to concatenate different lists This... is correct, but doesn't sound like a question. Is it meant to be? > 2. Do I need a response at all? No. You can tell AMP not to bother generating the protocol-level response by setting the requiresAnswer flag on your Command to False: <https://twistedmatrix.com/documents/15.4.0/api/twisted.protocols.amp.Command.html#requiresAnswer> > 3. Should I attempt to plug as many list items as possible into each > page (requires length checking of json-encoded strings and repeated > encoding/checks) or just choose a suitable limit of list items (my > current max length is about 200 characters and average is 71) of maybe > 300 list items per message? My current list is about 1k items in all, > and it's only going to get bigger. Why are you encoding as _both_ JSON and AMP? I'd say you should do the length-checking, because you still might end up with list items that are larger than expected if they're variable size. > 4. I was intrigued by the mention of 'Tubes' in the link above. Found > it here - https://tubes.readthedocs.org/en/latest/tube.html - should I > be using that instead? I'm writing a homegrown app which will only > really need (at this point) to communicate with itself and copies of > itself, and settled with AMP as being a simple way of achieving that. I would love it if you would help me test out and develop Tubes. If it is a small homegrown app it might be a good use-case. There are pros and cons: Tubes has higher test coverage and cleaner code since it was developed much more recently; but, it still has very limited functionality, badly broken areas, and no compatibility guarantees, because it's still somewhat experimental. However, Tubes is a way of implementing protocols, whereas AMP is an implementation of a request/response protocol. If you use Tubes, you'll need to do an implementation of AMP (or something like it) in order to issue requests and give responses. If I were you, especially since you've already figured out paging, I would probably just stick with AMP and Twisted as-is. -glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
