High Traffic
Hello, I am using django 1.3 on apache 2.* on windows server 2008. mysql 5.* When i try to access a view that does this: def view(request): len(User.objects.all()) return HttpResponse() i have about 9K results on that. i am doing len and not .count, because i want to test the server. the server becomes reaally slow. i am trying to test how much weight my server could manage, but it seems that 10 parallel requests to that view, slows the response down to 15 seconds!! THIS IS just an example. i will soon manage many requests, maybe not to that view always, but if 10 requests slows the server down that much, what would 1000 requests do? Please help. i dont want to migrate fomr django. is it the data to object proccess? or mysql throughput? How can i make this to go to a normal speed? BTW, even without a view: only py files calling that len(User.objects.all()), slows it really down! * COUNTING THE RESULTS IS NOT THE POINT Yarden -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: High Traffic
using aws ec2 & rds. so rolling out that. i monitor every query for high performance. but Users all is very simple query. what do u think it could other than those? thanks for all the help On Monday, April 16, 2012 10:59:55 PM UTC+3, creecode wrote: > > Hello Yarden, > > On Monday, April 16, 2012 12:25:36 PM UTC-7, Yarden Sachs wrote: > > Hello, I am using django 1.3 on apache 2.* on windows server 2008. >> mysql 5.* >> >> When i try to access a view that does this: def view(request): >> len(User.objects.all()) return HttpResponse() >> >> i have about 9K results on that. i am doing len and not .count, >> because i want to test the server. >> >> the server becomes reaally slow. >> > > Are you running Django and MySQL on the same server box? If so then you > are possibly running into speed issues because of contention. There are > several Djagno configuration docs on the web that talk about using at least > one server for your app and another for your database. Another thing to do > is move static files onto their own sever. > > You might want to try django-debug-toolbar to examine your queries for > your webpages to see if you can do some optimization. I highly recommend > giving it a go! > > Of course there are many things one can do to make things faster but > you'll need to give us some more info about your setup before we can help > you more. > > Toodle-loo. > creecode > -- You received this message because you are subscribed to the Google Groups "Django users" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/gTlmPOgGsIkJ. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: High Traffic
i have 9000 rows. and using mod_wsgi on apache2. what am i doing wrong? i tried to switch to nginx+fcgi. and tried ubuntu srrver as well. same results. any ideas? On 16 באפר 2012 23:18, "andrea mucci" wrote: > hi Yarden > > how many users have you in your database? > you use mod_fcgi or mod_python? ( more recommendable will be mod_fcgi or > mod_fcgid ) > > Django is framework that suppert very very large traffic. > you can see an example here > > http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances-dozens-of > > so, the most important think is configure well all the different component > around your project > like Apache2 ( big monster ), mod_fcgi etc…. > > cheers, > > > El 16/04/2012, a las 22:06, Yarden Sachs escribió: > > using aws ec2 & rds. so rolling out that. i monitor every query for high > performance. but Users all is very simple query. > > what do u think it could other than those? > > thanks for all the help > > > On Monday, April 16, 2012 10:59:55 PM UTC+3, creecode wrote: >> >> Hello Yarden, >> >> On Monday, April 16, 2012 12:25:36 PM UTC-7, Yarden Sachs wrote: >> >> Hello, I am using django 1.3 on apache 2.* on windows server 2008. >>> mysql 5.* >>> >>> When i try to access a view that does this: def view(request): >>> len(User.objects.all()) return HttpResponse() >>> >>> i have about 9K results on that. i am doing len and not .count, >>> because i want to test the server. >>> >>> the server becomes reaally slow. >>> >> >> Are you running Django and MySQL on the same server box? If so then you >> are possibly running into speed issues because of contention. There are >> several Djagno configuration docs on the web that talk about using at least >> one server for your app and another for your database. Another thing to do >> is move static files onto their own sever. >> >> You might want to try django-debug-toolbar to examine your queries for >> your webpages to see if you can do some optimization. I highly recommend >> giving it a go! >> >> Of course there are many things one can do to make things faster but >> you'll need to give us some more info about your setup before we can help >> you more. >> >> Toodle-loo**. >> creecode >> > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/django-users/-/gTlmPOgGsIkJ. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: High Traffic
ofcourse... i am using django since dec 10, and now i dont know what to do On 16 באפר 2012 23:30, "Ramiro Morales" wrote: > On Mon, Apr 16, 2012 at 5:22 PM, Yarden Sachs wrote: > > i have 9000 rows. and using mod_wsgi on apache2. what am i doing wrong? > i > > tried to switch to nginx+fcgi. and tried ubuntu srrver as well. same > > results. > > > > any ideas? > > Do you have DEBUG=False in your settings? > > -- > Ramiro Morales > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: High Traffic
what perpesfully used. for heavy system performance On 16 באפר 2012 23:38, "andrea mucci" wrote: > have you debugged the mysql query > > nginx+fgi work pretty well so is very very strange. > > have you tried to test, other type of query result without len method? > > > El 16/04/2012, a las 22:22, Yarden Sachs escribió: > > i have 9000 rows. and using mod_wsgi on apache2. what am i doing wrong? i > tried to switch to nginx+fcgi. and tried ubuntu srrver as well. same > results. > > any ideas? > On 16 באפר 2012 23:18, "andrea mucci" wrote: > >> hi Yarden >> >> how many users have you in your database? >> you use mod_fcgi or mod_python? ( more recommendable will be mod_fcgi or >> mod_fcgid ) >> >> Django is framework that suppert very very large traffic. >> you can see an example here >> >> http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances-dozens-of >> >> so, the most important think is configure well all the different >> component around your project >> like Apache2 ( big monster ), mod_fcgi etc…. >> >> cheers, >> >> >> El 16/04/2012, a las 22:06, Yarden Sachs escribió: >> >> using aws ec2 & rds. so rolling out that. i monitor every query for high >> performance. but Users all is very simple query. >> >> what do u think it could other than those? >> >> thanks for all the help >> >> >> On Monday, April 16, 2012 10:59:55 PM UTC+3, creecode wrote: >>> >>> Hello Yarden, >>> >>> On Monday, April 16, 2012 12:25:36 PM UTC-7, Yarden Sachs wrote: >>> >>> Hello, I am using django 1.3 on apache 2.* on windows server 2008. >>>> mysql 5.* >>>> >>>> When i try to access a view that does this: def view(request): >>>> len(User.objects.all()) return HttpResponse() >>>> >>>> i have about 9K results on that. i am doing len and not .count, >>>> because i want to test the server. >>>> >>>> the server becomes reaally slow. >>>> >>> >>> Are you running Django and MySQL on the same server box? If so then you >>> are possibly running into speed issues because of contention. There are >>> several Djagno configuration docs on the web that talk about using at least >>> one server for your app and another for your database. Another thing to do >>> is move static files onto their own sever. >>> >>> You might want to try django-debug-toolbar to examine your queries for >>> your webpages to see if you can do some optimization. I highly recommend >>> giving it a go! >>> >>> Of course there are many things one can do to make things faster but >>> you'll need to give us some more info about your setup before we can help >>> you more. >>> >>> Toodle-loo**. >>> creecode >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/django-users/-/gTlmPOgGsIkJ. >> To post to this group, send email to django-users@googlegroups.com. >> To unsubscribe from this group, send email to >> django-users+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/django-users?hl=en. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To post to this group, send email to django-users@googlegroups.com. >> To unsubscribe from this group, send email to >> django-users+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/django-users?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: High Traffic
Sure, I experienced unknown performance issues(slow responses) when high traffic would hit that site. so, started to test what happens if i hit the web site with a big calculation, like len function on an 9k objects. when i try to do it from client, it takes 0.7 secs to come back with response, if try to do it from 10 parralel client, it takes 15 secs for one http response! my team member, tried to find out the problem, and his conclusion is - python+django is not fast. and i DONT like that. its not right! So i am trying to find out why is that. if a i am trying to do simpler tasks like empty template rendering. or .count method, things go faster. but apart from the len function, i still performance issues if many users try to access the web site. its a delivery service application. Any ideas? On Monday, April 16, 2012 11:39:55 PM UTC+3, Javier Guerra wrote: > > On Mon, Apr 16, 2012 at 2:25 PM, Yarden Sachs wrote: > > * COUNTING THE RESULTS IS NOT THE POINT > > ok, then that len(User.objects.all()) is not an appropriate benchmark, > right? > > what you're doing is a view that fetches all 9K records from the DB, > without any filtering. will your views do that? usually you should > filter in the DB, where it can benefit from any indexes, and process > only those records that are relevant for that view. > > if you _have_ to do heavy processes to big portions of the data, then > you should do some denormalization, preprocessing, or queue > management. > > why don't you explain what do you want to achieve? > > -- > Javier > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/2Qys92uLTawJ. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: High Traffic
it took 20 times more for EACH request. that means that if 20 users accross the web ask for differrent heavy data- instead of taking a second for each one, every user will wait for 20 seconds before the response will answer. On 16 באפר 2012 23:54, "Javier Guerra Giraldez" wrote: > On Mon, Apr 16, 2012 at 3:49 PM, Yarden Sachs wrote: > > when i try to do it from client, it takes 0.7 secs to come back with > > response, if try to do it from 10 parralel client, it takes 15 secs for > one > > http response! > > > well, if you do a heavy and memory intensive task, then doing it ten > times is ten times as much process and ten times as much memory. i > think it's ok to take 20 times as much time. > > > > my team member, tried to find out the problem, and his conclusion is - > > python+django is not fast. and i DONT like that. its not right! > > no, the conclusion is that doing slow things is not fast > > -- > Javier > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: High Traffic
thanks! have any advice for good performance? and profiling? On 17 באפר 2012 00:18, "CLIFFORD ILKAY" wrote: > On 04/16/2012 04:49 PM, Yarden Sachs wrote: > >> Sure, >> >> I experienced unknown performance issues(slow responses) when high >> traffic would hit that site. >> > > > >> so, started to test what happens if i hit the web site with a big >> calculation, like len function on an 9k objects. when i try to do it >> from client, it takes 0.7 secs to come back with response, if try to >> do it from 10 parralel client, it takes 15 secs for one http >> response! >> > > If you're using MyISAM tables, that's a big performance killer. Every new > or expired session is going to get an exclusive table lock on the session > table. That will block reads for everyone else until the new row is > inserted. In the meantime, Apache is either forking and spawning new > threads, and you're creating more MySQL connections, which in turn is > consuming more RAM. Eventually, you'll run out of RAM and start hitting > swap/virtual memory. That only makes things worse. You have a contrived, > long-running query that only exacerbates the problem because while that > query is running, inserts or updates have to wait. All you've proved is > that you can create a cascade effect, which is hardly surprising. > > my team member, tried to find out the problem, and his conclusion is >> - python+django is not fast. and i DONT like that. its not right! >> > > I don't see how you can jump to that conclusion based on the evidence you > presented. Django powers some very busy sites on quite modest hardware > comparatively speaking but Django isn't going to make inefficient code or > poor architectural choices, like MyISAM tables, magically better. You need > to do some real profiling, not some contrived test that proves that running > slow queries is, well... slow. > -- > Regards, > > Clifford Ilkay > Dinamis > 1419-3230 Yonge St. > Toronto, ON > Canada M4N 3P6 > > <http://dinamis.com> > +1 416-410-3326 > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to django-users+unsubscribe@** > googlegroups.com . > For more options, visit this group at http://groups.google.com/** > group/django-users?hl=en<http://groups.google.com/group/django-users?hl=en> > . > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Special SELECT and UPDATE for a field
Hello,
i want to create a field class for geometry (not using geodjango).
here's the field code: http://pastebin.com/yADpZykJ
i want 2 things i can't seem to do:
1. I want the select to wrap the column with MySql AsText() Function
e.x. SELECT AsText(location) FROM addresses
2. I want to be able to save the value wrapped with a function e.x.
GeomFromText('POINT(0.0 0.0)')
but is seems that django will wrap that function with quotes, and so mysql
thinks its a string.
I found that in order to fix these, i will have to write an SQLCompiler,
Query, QuerySet, QuerySetManager. (or modify the get_columns and
get_default_columns methods of the SQLCompiler)
I am sure there is a simpler way.
What can I do?
Thanks!
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Special SELECT and UPDATE for a field
Its a hack. but it solves the save problem:
class CoordinatesQuerySetManager(models.Manager):
def __getattr__(self, name):
return getattr(self.get_query_set(), name)
def get_query_set(self):
q = self.model.QuerySet(self.model)
connection =
q.query.get_compiler(using=self.db).connection.cursor().cursor.cursor.connection
if not Coordinates in connection.encoders:
connection.encoders[Coordinates] = to_raw_sql
return q
i still cant do the select astext()
please HELP!
On Wednesday, January 30, 2013 10:14:57 PM UTC+2, Yarden Sachs wrote:
>
> Hello,
> i want to create a field class for geometry (not using geodjango).
> here's the field code: http://pastebin.com/yADpZykJ
>
> i want 2 things i can't seem to do:
> 1. I want the select to wrap the column with MySql AsText() Function
> e.x. SELECT AsText(location) FROM addresses
>
> 2. I want to be able to save the value wrapped with a function e.x.
> GeomFromText('POINT(0.0 0.0)')
> but is seems that django will wrap that function with quotes, and so mysql
> thinks its a string.
>
> I found that in order to fix these, i will have to write an SQLCompiler,
> Query, QuerySet, QuerySetManager. (or modify the get_columns and
> get_default_columns methods of the SQLCompiler)
> I am sure there is a simpler way.
>
> What can I do?
>
> Thanks!
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Special SELECT and UPDATE for a field
You can see in my paste that i already created a custom field for geometry
field.
On Thu, Jan 31, 2013 at 7:52 PM, Tom Evans wrote:
> On Wed, Jan 30, 2013 at 8:14 PM, Yarden Sachs wrote:
> > Hello,
> > i want to create a field class for geometry (not using geodjango).
> > here's the field code: http://pastebin.com/yADpZykJ
> >
> > i want 2 things i can't seem to do:
> > 1. I want the select to wrap the column with MySql AsText() Function
> > e.x. SELECT AsText(location) FROM addresses
> >
> > 2. I want to be able to save the value wrapped with a function e.x.
> > GeomFromText('POINT(0.0 0.0)')
> > but is seems that django will wrap that function with quotes, and so
> mysql
> > thinks its a string.
> >
> > I found that in order to fix these, i will have to write an SQLCompiler,
> > Query, QuerySet, QuerySetManager. (or modify the get_columns and
> > get_default_columns methods of the SQLCompiler)
> > I am sure there is a simpler way.
> >
> > What can I do?
> >
> > Thanks!
>
> Sounds like you want to define a custom field rather than fiddle with
> a Model's manager.
>
> https://docs.djangoproject.com/en/1.4/howto/custom-model-fields/
>
> Cheers
>
> Tom
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Special SELECT and UPDATE for a field
In the end,
what i did is, i added a conversion type to MySQLDB for my Corrdinates
object to convert it to a non literal string.
and for the selecting part, i had to copy and alter db/sql/compiler's
get_columns and get_default_columns to wrap my field selection with the
AsText function. a very bad solution. but it worked.
this was really difficult to do, there is no reason for django not to let
me tell django to put in NOW() or AsText or any non literal string.
On Wednesday, January 30, 2013 10:14:57 PM UTC+2, Yarden Sachs wrote:
>
> Hello,
> i want to create a field class for geometry (not using geodjango).
> here's the field code: http://pastebin.com/yADpZykJ
>
> i want 2 things i can't seem to do:
> 1. I want the select to wrap the column with MySql AsText() Function
> e.x. SELECT AsText(location) FROM addresses
>
> 2. I want to be able to save the value wrapped with a function e.x.
> GeomFromText('POINT(0.0 0.0)')
> but is seems that django will wrap that function with quotes, and so mysql
> thinks its a string.
>
> I found that in order to fix these, i will have to write an SQLCompiler,
> Query, QuerySet, QuerySetManager. (or modify the get_columns and
> get_default_columns methods of the SQLCompiler)
> I am sure there is a simpler way.
>
> What can I do?
>
> Thanks!
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
