On Thu, 25 Feb 2010 15:29:34 +0000 "Martin P. Hellwig" <martin.hell...@dcuktec.org> wrote: > On 02/25/10 13:58, D'Arcy J.M. Cain wrote: > > On Thu, 25 Feb 2010 02:26:18 -0800 (PST) > <cut> > > Our biggest problem was in > > a network heavy element of the app and that was low level TCP/IP stuff > > that rather than being Python's problem was something we used Python to > > fix. > <cut> > Out off interest, could you elaborate on that?
Somewhat - there is an NDA so I can't give exact details. It was crucial to our app that we sync up databases in Canada and the US (later Britain, Europe and Japan) in real time with those transactions. Our problem was that even though our two server systems were on the backbone, indeed with the same major carrier, we could not keep them in sync. We were taking way to long to transact an update across the network. The problem had to do with the way TCP/IP works, especially closer to the core. Our provider was collecting data and sending it only after filling a buffer or after a timeout. The timeout was short so it wouldn't normally be noticed and in most cases (web pages e.g.) the connection is opened, data is pushed and the connection is closed so the buffer is flushed immediately. Our patterns were different so we were hitting the timeout on every single transaction and there was no way we would have been able to keep up. Our first crack at fixing this was to simply add garbage to the packet we were sending. Making the packets an order of magnitude bigger sped up the proccessing dramatically. That wasn't a very clean solution though so we looked for a better way. That better way turned out to asynchronous update transactions. All we did was keep feeding updates to the remote site and forget about ACKS. We then had a second process which handled ACKS and tracked which packets had been properly transferred. The system had IDs on each update and retries happened if ACKS didn't happen soon enough. Naturally we ignored ACKS that we had already processed. All of the above (and much more complexity not even discussed here) was handled by Python code and database manipulation. There were a few bumps along the way but overall it worked fine. If we were using C or even assembler we would not have sped up anything and the solution we came up with would have been horrendous to code. As it was I and my chief programmer locked ourselves in the boardroom and had a working solution before the day was out. Python wins again. -- D'Arcy J.M. Cain <da...@druid.net> | Democracy is three wolves http://www.druid.net/darcy/ | and a sheep voting on +1 416 425 1212 (DoD#0082) (eNTP) | what's for dinner. -- http://mail.python.org/mailman/listinfo/python-list
