taking python enterprise level?...
hello people, i have been reading posts on this group for quite some time now and many, if not all (actually not all!), seem quite interesting. i plan to build an application, a network based application that i estimate (and seriously hope) would get as many as 100, 000 hits a day (hehe,...my dad always told me to 'AIM HIGH' ;0), not some 'facebook' or anything like it, its mainly for a financial transactions which gets pretty busy... so my question is this would anyone have anything that would make python a little less of a serious candidate (cos it already is) and the options may be to use some other languages (maybe java, C (oh God))...i am into a bit of php and building API's in php would not be the hard part, what i am concerned about is scalability and efficiency, well, as far as the 'core' is concerned. would python be able to manage giving me a solid 'core' and will i be able to use python provide any API i would like to implement?... im sorry if my subject was not as clear as probably should be!. i guess this should be the best place to ask this sort of thing, hope im so right. Thanks -- http://mail.python.org/mailman/listinfo/python-list
Re: taking python enterprise level?...
On Feb 25, 12:13 pm, Steve Holden wrote: > simn_stv wrote: > > hello people, i have been reading posts on this group for quite some > > time now and many, if not all (actually not all!), seem quite > > interesting. > > i plan to build an application, a network based application that i > > estimate (and seriously hope) would get as many as 100, 000 hits a day > > (hehe,...my dad always told me to 'AIM HIGH' ;0), not some 'facebook' > > or anything like it, its mainly for a financial transactions which > > gets pretty busy... > > so my question is this would anyone have anything that would make > > python a little less of a serious candidate (cos it already is) and > > the options may be to use some other languages (maybe java, C (oh > > God))...i am into a bit of php and building API's in php would not be > > the hard part, what i am concerned about is scalability and > > efficiency, well, as far as the 'core' is concerned. > > > would python be able to manage giving me a solid 'core' and will i be > > able to use python provide any API i would like to implement?... > > > im sorry if my subject was not as clear as probably should be!. > > i guess this should be the best place to ask this sort of thing, hope > > im so right. > > > Thanks > > I'd suggest that if you are running an operation that gets 100,000 hits > a day then your problems won't be with Python but with organizational > aspects of your operation. > > regards > Steve > -- very well noted steve, i'd be careful (which is a very relative word) with the organizational aspects... i'm sure ure quite rooted in that aspect, hey u need a job??;) -- http://mail.python.org/mailman/listinfo/python-list
Re: taking python enterprise level?...
On Feb 25, 12:21 pm, "Martin P. Hellwig" wrote: > On 02/25/10 10:26, simn_stv wrote: > > what i am concerned about is scalability and > > efficiency, well, as far as the 'core' is concerned. > > > would python be able to manage giving me a solid 'core' and will i be > > able to use python provide any API i would like to implement?... > > > Python isn't the most efficient language, the assembler provided by the > maker of your CPU probably is the best you can get, LOL...;), yeah right, the mere thought of writing assembler instructions is SCARY!! >everything after > that is a trade-off between performance and flexibility (flexible in the > most flexible sense of the word :-)). > > That being said, for me, Python (well actually any turing complete > programming language), is more like a box of lego with infinite amount > of pieces. > Scalability and API issues are the same as the shape and function of the > model your making with lego. > > Sure some type of pieces might be more suited than other types but since > you can simulate any type of piece with the ones that are already > present, you are more limited by your imagination than by the language. > > So in short, I don't see any serious problems using Python, I have used > it in Enterprise environments without any problems but than again I was > aware not to use it for numerical intensive parts without the use of 3rd > party libraries like numpy. Which for me resulted in not doing the > compression of a database delta's in pure python but to offload that to > a more suitable external program, still controlled from Python though. > > -- > mph -- http://mail.python.org/mailman/listinfo/python-list
Re: taking python enterprise level?...
On Feb 25, 5:18 pm, "D'Arcy J.M. Cain" wrote: > On Thu, 25 Feb 2010 15:29:34 + > "Martin P. Hellwig" wrote: > > > On 02/25/10 13:58, D'Arcy J.M. Cain wrote: > > > On Thu, 25 Feb 2010 02:26:18 -0800 (PST) > > > > > Our biggest problem was in > > > a network heavy element of the app and that was low level TCP/IP stuff > > > that rather than being Python's problem was something we used Python to > > > fix. > > > > Out off interest, could you elaborate on that? > > Somewhat - there is an NDA so I can't give exact details. It was > crucial to our app that we sync up databases in Canada and the US (later > Britain, Europe and Japan) in real time with those transactions. Our > problem was that even though our two server systems were on the > backbone, indeed with the same major carrier, we could not keep them in > sync. We were taking way to long to transact an update across the > network. > > The problem had to do with the way TCP/IP works, especially closer to > the core. Our provider was collecting data and sending it only after > filling a buffer or after a timeout. The timeout was short so it > wouldn't normally be noticed and in most cases (web pages e.g.) the > connection is opened, data is pushed and the connection is closed so > the buffer is flushed immediately. Our patterns were different so we > were hitting the timeout on every single transaction and there was no > way we would have been able to keep up. > > Our first crack at fixing this was to simply add garbage to the packet > we were sending. Making the packets an order of magnitude bigger sped > up the proccessing dramatically. That wasn't a very clean solution > though so we looked for a better way. > > That better way turned out to asynchronous update transactions. All we > did was keep feeding updates to the remote site and forget about ACKS. > We then had a second process which handled ACKS and tracked which > packets had been properly transferred. The system had IDs on each > update and retries happened if ACKS didn't happen soon enough. > Naturally we ignored ACKS that we had already processed. > > All of the above (and much more complexity not even discussed here) was > handled by Python code and database manipulation. There were a few > bumps along the way but overall it worked fine. If we were using C or > even assembler we would not have sped up anything and the solution we > came up with would have been horrendous to code. As it was I and my > chief programmer locked ourselves in the boardroom and had a working > solution before the day was out. sure it wouldnt have sped it up a bit, even a bit?; probably the development and maintenance time would be a nightmare but it should speed the app up a bit... > > Python wins again. > > -- > D'Arcy J.M. Cain | Democracy is three > wolveshttp://www.druid.net/darcy/ | and a sheep voting on > +1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner. seriously added to the reputation of python, from my own perspective...kudos python! -- http://mail.python.org/mailman/listinfo/python-list
Re: taking python enterprise level?...
On Feb 25, 5:18 pm, "D'Arcy J.M. Cain" wrote: > On Thu, 25 Feb 2010 15:29:34 + > "Martin P. Hellwig" wrote: > > > On 02/25/10 13:58, D'Arcy J.M. Cain wrote: > > > On Thu, 25 Feb 2010 02:26:18 -0800 (PST) > > > > > Our biggest problem was in > > > a network heavy element of the app and that was low level TCP/IP stuff > > > that rather than being Python's problem was something we used Python to > > > fix. > > > > Out off interest, could you elaborate on that? > > Somewhat - there is an NDA so I can't give exact details. It was > crucial to our app that we sync up databases in Canada and the US (later > Britain, Europe and Japan) in real time with those transactions. Our > problem was that even though our two server systems were on the > backbone, indeed with the same major carrier, we could not keep them in > sync. We were taking way to long to transact an update across the > network. > > The problem had to do with the way TCP/IP works, especially closer to > the core. Our provider was collecting data and sending it only after > filling a buffer or after a timeout. The timeout was short so it > wouldn't normally be noticed and in most cases (web pages e.g.) the > connection is opened, data is pushed and the connection is closed so > the buffer is flushed immediately. Our patterns were different so we > were hitting the timeout on every single transaction and there was no > way we would have been able to keep up. > > Our first crack at fixing this was to simply add garbage to the packet > we were sending. Making the packets an order of magnitude bigger sped > up the proccessing dramatically. That wasn't a very clean solution > though so we looked for a better way. > > That better way turned out to asynchronous update transactions. All we > did was keep feeding updates to the remote site and forget about ACKS. > We then had a second process which handled ACKS and tracked which > packets had been properly transferred. The system had IDs on each > update and retries happened if ACKS didn't happen soon enough. > Naturally we ignored ACKS that we had already processed. > > All of the above (and much more complexity not even discussed here) was > handled by Python code and database manipulation. There were a few > bumps along the way but overall it worked fine. If we were using C or > even assembler we would not have sped up anything and the solution we > came up with would have been horrendous to code. As it was I and my > chief programmer locked ourselves in the boardroom and had a working > solution before the day was out. sure it wouldnt have sped it up a bit, even a bit?; probably the development and maintenance time would be a nightmare but it should speed the app up a bit... > > Python wins again. > > -- > D'Arcy J.M. Cain | Democracy is three > wolveshttp://www.druid.net/darcy/ | and a sheep voting on > +1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner. seriously added to the reputation of python, from my own perspective...kudos python! -- http://mail.python.org/mailman/listinfo/python-list
Re: taking python enterprise level?...
On Feb 26, 10:32 am, mdipierro wrote: > 100,000 hits a day is not a low. I get that some day on my web server > without problem and without one request dropped. > > Most frameworks web2py, Django, Pylons can handle that kind of load > since Python is not the bottle neck. taking a look at django right now, doesnt look too bad from where im standing, maybe when i get into the code i'd run into some issues that would cause some headaches!! > You have to follow some tricks: > > 1) have the web server serve static pages directly and set the pragma > cache expire to one month > 2) cache all pages that do not have forms for at least few minutes > 3) avoid database joins but this would probably be to the detriment of my database design, which is a no-no as far as im concerned. The way the tables would be structured requires 'joins' when querying the db; or could you elaborate a little?? > 4) use a server with at least 512KB Ram. hmmm...!, still thinking about what you mean by this statement also. > 5) if you pages are large, use gzip compression > > If you develop your app with the web2py framework, you always have the > option to deploy on the Google App Engine. If you can live with their > constraints you should have no scalability problems. > > Massimo > > On Feb 25, 4:26 am, simn_stv wrote: > > > hello people, i have been reading posts on this group for quite some > > time now and many, if not all (actually not all!), seem quite > > interesting. > > i plan to build an application, a network based application that i > > estimate (and seriously hope) would get as many as 100, 000 hits a day > > (hehe,...my dad always told me to 'AIM HIGH' ;0), not some 'facebook' > > or anything like it, its mainly for a financial transactions which > > gets pretty busy... > > so my question is this would anyone have anything that would make > > python a little less of a serious candidate (cos it already is) and > > the options may be to use some other languages (maybe java, C (oh > > God))...i am into a bit of php and building API's in php would not be > > the hard part, what i am concerned about is scalability and > > efficiency, well, as far as the 'core' is concerned. > > > would python be able to manage giving me a solid 'core' and will i be > > able to use python provide any API i would like to implement?... > > > im sorry if my subject was not as clear as probably should be!. > > i guess this should be the best place to ask this sort of thing, hope > > im so right. > > > Thanks thanks for the feedback... -- http://mail.python.org/mailman/listinfo/python-list
Re: taking python enterprise level?...
On Feb 26, 10:19 am, "Diez B. Roggisch" wrote: > Am 26.02.10 05:01, schrieb D'Arcy J.M. Cain: > > > > > On Fri, 26 Feb 2010 01:12:00 +0100 > > "Diez B. Roggisch" wrote: > >>> That better way turned out to asynchronous update transactions. All we > >>> did was keep feeding updates to the remote site and forget about ACKS. > >>> We then had a second process which handled ACKS and tracked which > >>> packets had been properly transferred. The system had IDs on each > >>> update and retries happened if ACKS didn't happen soon enough. > >>> Naturally we ignored ACKS that we had already processed. > > >> sounds like using UDP to me, of course with a protocol on top (namely > >> the one you implemented). > > >> Any reason you sticked to TCP instead? > > > TCP does a great job of delivering a stream of data in order and > > handling the retries. The app really was connection oriented and we > > saw no reason to emulate that over an unconnected protocol. There were > > other wheels to reinvent that were more important. > > So when you talk about ACKs, you don't mean these on the TCP-level > (darn, whatever iso-level that is...), but on some higher level? > > Diez i think its on the TCP that he's referring to or is it?... if it is, that means he's doing some 'mean' network level scripting, impressive...but i never thought python could go that deep in network programming!... -- http://mail.python.org/mailman/listinfo/python-list
Re: taking python enterprise level?...
till i think i absolutely need to trade-off easier and less complicated code, better db structure (from a relational perspective) and generally less "head aches" for speed, i think i'll stick with the joins for now!...;) the thought of denormalization really doesnt appeal to me... -- http://mail.python.org/mailman/listinfo/python-list
