On Feb 25, 5:18 pm, "D'Arcy J.M. Cain" <da...@druid.net> wrote: > On Thu, 25 Feb 2010 15:29:34 +0000 > "Martin P. Hellwig" <martin.hell...@dcuktec.org> wrote: > > > On 02/25/10 13:58, D'Arcy J.M. Cain wrote: > > > On Thu, 25 Feb 2010 02:26:18 -0800 (PST) > > <cut> > > > Our biggest problem was in > > > a network heavy element of the app and that was low level TCP/IP stuff > > > that rather than being Python's problem was something we used Python to > > > fix. > > <cut> > > Out off interest, could you elaborate on that? > > Somewhat - there is an NDA so I can't give exact details. It was > crucial to our app that we sync up databases in Canada and the US (later > Britain, Europe and Japan) in real time with those transactions. Our > problem was that even though our two server systems were on the > backbone, indeed with the same major carrier, we could not keep them in > sync. We were taking way to long to transact an update across the > network. > > The problem had to do with the way TCP/IP works, especially closer to > the core. Our provider was collecting data and sending it only after > filling a buffer or after a timeout. The timeout was short so it > wouldn't normally be noticed and in most cases (web pages e.g.) the > connection is opened, data is pushed and the connection is closed so > the buffer is flushed immediately. Our patterns were different so we > were hitting the timeout on every single transaction and there was no > way we would have been able to keep up. > > Our first crack at fixing this was to simply add garbage to the packet > we were sending. Making the packets an order of magnitude bigger sped > up the proccessing dramatically. That wasn't a very clean solution > though so we looked for a better way. > > That better way turned out to asynchronous update transactions. All we > did was keep feeding updates to the remote site and forget about ACKS. > We then had a second process which handled ACKS and tracked which > packets had been properly transferred. The system had IDs on each > update and retries happened if ACKS didn't happen soon enough. > Naturally we ignored ACKS that we had already processed. > > All of the above (and much more complexity not even discussed here) was > handled by Python code and database manipulation. There were a few > bumps along the way but overall it worked fine. If we were using C or > even assembler we would not have sped up anything and the solution we > came up with would have been horrendous to code. As it was I and my > chief programmer locked ourselves in the boardroom and had a working > solution before the day was out.
sure it wouldnt have sped it up a bit, even a bit?; probably the development and maintenance time would be a nightmare but it should speed the app up a bit... > > Python wins again. > > -- > D'Arcy J.M. Cain <da...@druid.net> | Democracy is three > wolveshttp://www.druid.net/darcy/ | and a sheep voting on > +1 416 425 1212 (DoD#0082) (eNTP) | what's for dinner. seriously added to the reputation of python, from my own perspective...kudos python! -- http://mail.python.org/mailman/listinfo/python-list
