Re: How to know our Django version

2016-02-03 Thread communication
Hello, 

and if not ? 

i have actually only acces to the web part.

Le mardi 2 février 2016 18:43:56 UTC+1, James Bennett a écrit :
>
> As long as you have access to a shell, you can do
>
> python manage.py shell
>
> Then in the interpreter, do
>
> import django
> print(django.get_version())
>
> On Tue, Feb 2, 2016 at 6:59 AM,  > wrote:
>
>> Hello all, 
>>
>> thanks in advance for your help. 
>>
>> i'm just integrating new office and they actually have a website based on 
>> Django. 
>>
>> but we have no more contact with the web agency who developp this site. 
>>
>> So i wanted to know what is the version of Django used by our site ? 
>>
>> Could you please help me ?
>>
>> Best regards
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com .
>> To post to this group, send email to django...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/c5946709-5f82-4caa-bf68-aa59a24e1a6b%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/fe2b4c48-3bd9-4c5b-b0a6-52f177d99621%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re:Re: How to know our Django version

2016-02-03 Thread 林攀
django --version





--

林攀

在 2016-02-03 19:05:01,communicat...@domainedemanville.fr 写道:

Hello, 


and if not ? 


i have actually only acces to the web part.

Le mardi 2 février 2016 18:43:56 UTC+1, James Bennett a écrit :
As long as you have access to a shell, you can do


python manage.py shell


Then in the interpreter, do


import django
print(django.get_version())


On Tue, Feb 2, 2016 at 6:59 AM,  wrote:

Hello all, 


thanks in advance for your help. 


i'm just integrating new office and they actually have a website based on 
Django. 


but we have no more contact with the web agency who developp this site. 


So i wanted to know what is the version of Django used by our site ? 


Could you please help me ?


Best regards

--
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/c5946709-5f82-4caa-bf68-aa59a24e1a6b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.




--
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/fe2b4c48-3bd9-4c5b-b0a6-52f177d99621%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/289ed9df.2b10.152a74f7f53.Coremail.18610710105%40163.com.
For more options, visit https://groups.google.com/d/optout.


Provide me Links to learn django framework on Linux

2016-02-03 Thread Ajay Sharma
Hi everyone on the Group. 

I want resources and web links, where i can learn  Django Framework On 
Linux platform as I 

am using Ubuntu for the development. 

Please help me . 

thank you to every one in Advance. 

>From Ajay .  :) 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/4386ebcc-6ebd-4689-9f0b-a4496edaa41b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


ProgrammingError django_content_type already exists

2016-02-03 Thread M Hashmi
I've been stuck on this issue for a while now. If anyone have concrete 
solution please let me know.
I subscribed for a VPS on DigitalOcean.com Ubuntu with OneClickDjango 
installation. Pre-loaded installation is 1.6.1 and when I upgrade it I get 
Programming Error django_content_type already exists. No matter what I do 
but issue still persists. 

Please help.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/1416e5e6-e90e-41e7-8107-0ec6fea7ce1a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[no subject]

2016-02-03 Thread Denis
Hi,

I'm looking at upgrading my application from 1.6 to 1.9.2... An issue I'm
encountering now is that on Heroku, running collectstatic doesn't work on
1.9.2. The odd  things is that if I try 1.8.7 instead, it work great...


Running:
remote: python manage.py collectstatic -i docs -i tests --noinput -v 3
remote: Copying '/app/accounts/static/...js'
remote: Copying '/app/accounts/static/js'
remote: Copying '/app/accounts/static/js'
... [ Thousands of lines... ]
remote: Found another file with the destination path
'app/controllers/...js'. It will be ignored since only the first
encountered file is collected. If this is not what you want, make sure
every static file has a unique path.
remote: Found another file with the destination path
'app/controllers/js'. It will be ignored since only the first
encountered file is collected. If this is not what you want, make sure
every static file has a unique path.
remote: Traceback (most recent call last):
remote: output = self.handle(*args, **options)
remote:   File
"/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py",
line 176, in handle
remote: collected = self.collect()
remote:   File
"/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py",
line 114, in collect
remote: level=1,
remote:   File
"/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py",
line 201, in log
remote: self.stdout.write(msg)
remote:   File
"/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py",
line 111, in write
remote: self._out.write(force_str(style_func(msg)))
remote: IOError: [Errno 11] Resource temporarily unavailable


I do have a lot of static file... Unsure that matter.

An other things is that it doesn't seem to fail at the same point in the
collectstatic command every time I run it. So look like it's failing on
different file every time I run it...


I did find this: https://github.com/etianen/django-require/issues/20
But I actually don't use django-require... (It's not in my requirement.txt
of what I install on Heroku)


I also see it fail (sometime but much less often as the "Resource
temporarily unavaible" error this way:
remote: Found another file with the destination path 'accounts/..html'. It
will be ignored since only the first encountered file is collected. If this
is not what you want, make sure every static file has a unique path.
remote: Found another file with the destination path 'accounts/...js'. It
will be ignored since only the first encountered file is collected. If this
is not what you want, make sure every static file has a unique path.
remote: Traceback (most recent call last):
remote:   File "manage.py", line 10, in 
remote: execute_from_command_line(sys.argv)
remote: utility.execute()
remote: self.execute(*args, **cmd_options)

Which is even less helpful in figuring out what's going on...


So I'm looking for some though on what this error mean and how to debug it
further. We never encountered that error when on 1.6. Testing on 1.8.7
works fine but then if I only change the Django version (nothing else) to
1.9.2, I start getting that error...


Thanks everyone!

Denis Bellavance
Peach
Seattle

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAOdjZ6K7j9Xv70%2BRYSBEdFom-VWRLMMy_e2CRZzfqmngA1ZBrw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: How to know our Django version

2016-02-03 Thread Tomas Tombakas
If you don't have access to the shell, why are you concerned with the
django version?

On 3 February 2016 at 13:05,  wrote:

> Hello,
>
> and if not ?
>
> i have actually only acces to the web part.
>
> Le mardi 2 février 2016 18:43:56 UTC+1, James Bennett a écrit :
>>
>> As long as you have access to a shell, you can do
>>
>> python manage.py shell
>>
>> Then in the interpreter, do
>>
>> import django
>> print(django.get_version())
>>
>> On Tue, Feb 2, 2016 at 6:59 AM,  wrote:
>>
>>> Hello all,
>>>
>>> thanks in advance for your help.
>>>
>>> i'm just integrating new office and they actually have a website based
>>> on Django.
>>>
>>> but we have no more contact with the web agency who developp this site.
>>>
>>> So i wanted to know what is the version of Django used by our site ?
>>>
>>> Could you please help me ?
>>>
>>> Best regards
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users...@googlegroups.com.
>>> To post to this group, send email to django...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-users.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-users/c5946709-5f82-4caa-bf68-aa59a24e1a6b%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/fe2b4c48-3bd9-4c5b-b0a6-52f177d99621%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHsc%3DGNqcYSeYzZgZG33oKYEQHwC8xJazXw275_Su9sht909sg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Django getting slow with uwsgi

2016-02-03 Thread Arink Verma


Hi Django Experts.

I have recently migrated my Django app to Django 1.8.8 since then I am 
experiencing low performance.

With cProfiling I am have generated following breakdown of time spent.

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   49   22.0430.450   22.0440.450 {select.select}
8   10.1151.264   10.1151.264 {method 'recv' of 
'_socket.socket' objects}
   160.0880.0050.0880.005 {method 'read' of 'file' objects}
 1513/3830.0520.0000.1550.000 
/usr/lib/python2.7/sre_parse.py:379(_parse)
294690.0410.0000.0510.000 
/usr/lib/python2.7/sre_parse.py:182(__next)
   250.0390.0020.0440.002 
/usr/lib/python2.7/sre_compile.py:301(_optimize_unicode)
 2707/3580.0240.0000.1200.000 
/usr/lib/python2.7/sre_compile.py:32(_compile)
  9090.0230.0000.0810.000 
/usr/lib/python2.7/sre_compile.py:207(_optimize_charset)
   170.0200.0010.0200.001 {_socket.gethostbyaddr}
50.0170.0030.0170.003 {_ctypes.dlopen}
254820.0160.0000.0590.000 
/usr/lib/python2.7/sre_parse.py:201(get)
  3810.0150.0000.0170.000 
/usr/local/lib/python2.7/dist-packages/django/db/models/options.py:711(_expire_cache)
   290.0150.0010.0180.001 
/usr/lib/python2.7/collections.py:237(namedtuple)
77283/759870.0140.0000.0140.000 {len}
3603/13850.0140.0000.0160.000 
/usr/lib/python2.7/sre_parse.py:140(getwidth)
  4530.0140.0000.0270.000 
/usr/local/lib/python2.7/dist-packages/django/utils/functional.py:102(__prepare_class__)
413320.0130.0000.0130.000 {hasattr}
  227/1630.0120.0000.6650.004 {__import__}
   160.0120.0010.0120.001 {posix.popen}
525220.0110.0000.0110.000 {method 'append' of 'list' 
objects}

How can I indefinitely the bottleneck whether it is in uwsgi's config or 
Django?
uWsgi App config


python
/run/uwsgi/app/api.example.com/api.example.com.socket
/home/ubuntu/uwsgis/

wsgi_configuration_module


2
true
60
8
1
/tmp/stats.socket
2
2048
256
192
application
/home/ubuntu/uwsgi_core.log
/home/ubuntu/uwsgi_core_daemonize.log





-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/d9d76d82-c4c0-4348-8522-12e5c5f3833f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Provide me Links to learn django framework on Linux

2016-02-03 Thread Rafael E. Ferrero
For Python
https://www.codecademy.com/learn/python
https://www.hackerrank.com/domains/python/py-introduction

For Django
https://docs.djangoproject.com/en/1.9/



Rafael E. Ferrero

2016-02-03 8:30 GMT-03:00 Ajay Sharma :

> Hi everyone on the Group.
>
> I want resources and web links, where i can learn  Django Framework On
> Linux platform as I
>
> am using Ubuntu for the development.
>
> Please help me .
>
> thank you to every one in Advance.
>
> From Ajay .  :)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/4386ebcc-6ebd-4689-9f0b-a4496edaa41b%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAJJc_8XSoLXCVpumKfE%2BpxJ7XuG2PDp1Jn0VwOZbko04TygZzw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re:

2016-02-03 Thread Sergiy Khohlov
 Hello Denis,

 This issue  is not related  to the  django-require.  Look like you have
two  files (with  same name ) at the different paths. New  collect static
is trying to colelct both files and can not decided  which one should be
used.
Could you please  send part of the section  STATICFILES_DIRS and
INSTALLED_APP ?

Many thanks,

Serge


+380 636150445
skype: skhohlov

On Wed, Feb 3, 2016 at 9:19 AM, Denis  wrote:

> Hi,
>
> I'm looking at upgrading my application from 1.6 to 1.9.2... An issue I'm
> encountering now is that on Heroku, running collectstatic doesn't work on
> 1.9.2. The odd  things is that if I try 1.8.7 instead, it work great...
>
>
> Running:
> remote: python manage.py collectstatic -i docs -i tests --noinput -v 3
> remote: Copying '/app/accounts/static/...js'
> remote: Copying '/app/accounts/static/js'
> remote: Copying '/app/accounts/static/js'
> ... [ Thousands of lines... ]
> remote: Found another file with the destination path
> 'app/controllers/...js'. It will be ignored since only the first
> encountered file is collected. If this is not what you want, make sure
> every static file has a unique path.
> remote: Found another file with the destination path
> 'app/controllers/js'. It will be ignored since only the first
> encountered file is collected. If this is not what you want, make sure
> every static file has a unique path.
> remote: Traceback (most recent call last):
> remote: output = self.handle(*args, **options)
> remote:   File
> "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py",
> line 176, in handle
> remote: collected = self.collect()
> remote:   File
> "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py",
> line 114, in collect
> remote: level=1,
> remote:   File
> "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py",
> line 201, in log
> remote: self.stdout.write(msg)
> remote:   File
> "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py",
> line 111, in write
> remote: self._out.write(force_str(style_func(msg)))
> remote: IOError: [Errno 11] Resource temporarily unavailable
>
>
> I do have a lot of static file... Unsure that matter.
>
> An other things is that it doesn't seem to fail at the same point in the
> collectstatic command every time I run it. So look like it's failing on
> different file every time I run it...
>
>
> I did find this: https://github.com/etianen/django-require/issues/20
> But I actually don't use django-require... (It's not in my requirement.txt
> of what I install on Heroku)
>
>
> I also see it fail (sometime but much less often as the "Resource
> temporarily unavaible" error this way:
> remote: Found another file with the destination path 'accounts/..html'. It
> will be ignored since only the first encountered file is collected. If this
> is not what you want, make sure every static file has a unique path.
> remote: Found another file with the destination path 'accounts/...js'. It
> will be ignored since only the first encountered file is collected. If this
> is not what you want, make sure every static file has a unique path.
> remote: Traceback (most recent call last):
> remote:   File "manage.py", line 10, in 
> remote: execute_from_command_line(sys.argv)
> remote: utility.execute()
> remote: self.execute(*args, **cmd_options)
>
> Which is even less helpful in figuring out what's going on...
>
>
> So I'm looking for some though on what this error mean and how to debug it
> further. We never encountered that error when on 1.6. Testing on 1.8.7
> works fine but then if I only change the Django version (nothing else) to
> 1.9.2, I start getting that error...
>
>
> Thanks everyone!
>
> Denis Bellavance
> Peach
> Seattle
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAOdjZ6K7j9Xv70%2BRYSBEdFom-VWRLMMy_e2CRZzfqmngA1ZBrw%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this grou

Re: Provide me Links to learn django framework on Linux

2016-02-03 Thread Christian Ledermann
http://chimera.labs.oreilly.com/books/123400754/index.html

On 3 February 2016 at 14:33, Rafael E. Ferrero  wrote:
> For Python
> https://www.codecademy.com/learn/python
> https://www.hackerrank.com/domains/python/py-introduction
>
> For Django
> https://docs.djangoproject.com/en/1.9/
>
>
>
> Rafael E. Ferrero
>
> 2016-02-03 8:30 GMT-03:00 Ajay Sharma :
>>
>> Hi everyone on the Group.
>>
>> I want resources and web links, where i can learn  Django Framework On
>> Linux platform as I
>>
>> am using Ubuntu for the development.
>>
>> Please help me .
>>
>> thank you to every one in Advance.
>>
>> From Ajay .  :)
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-users@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/4386ebcc-6ebd-4689-9f0b-a4496edaa41b%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAJJc_8XSoLXCVpumKfE%2BpxJ7XuG2PDp1Jn0VwOZbko04TygZzw%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.



-- 
Best Regards,

Christian Ledermann

Newark-on-Trent - UK
Mobile : +44 7474997517

https://uk.linkedin.com/in/christianledermann
https://github.com/cleder/


<*)))>{

If you save the living environment, the biodiversity that we have left,
you will also automatically save the physical environment, too. But If
you only save the physical environment, you will ultimately lose both.

1) Don’t drive species to extinction

2) Don’t destroy a habitat that species rely on.

3) Don’t change the climate in ways that will result in the above.

}<(((*>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CABCjzWrA94yfJGM_kHDAzz_F%3D%3DaKuASN3mcfYV4NjXdsfRnjWg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Scaling Django

2016-02-03 Thread Joshua Pokotilow
At the startup where I work, we've written a lot of our server code in 
Django. So far, we've adopted a "build it fast" mentality, so we invested 
very little time in optimizing our code. A small amount of load testing has 
revealed our codebase / infrastructure as it stands today needs to run 
faster and support more users.

We recently hired some new engineers who are extremely skeptical that we 
should optimize our existing code. Their main concerns are:

- We need to move to a service-oriented infrastructure because Django is 
too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
- It's too easy to write slow queries using the Django ORM
- It's hard to hire Django engineers
- While Instagram and DISQUS use Django to service large numbers of people, 
they don't use it for any serious backend work

After having worked with Django for the last 3 years, I'm a big believer in 
it, and I believe it would scale. To defend my position, I've pointed out 
to my colleagues that it's easy to identify bottlenecks with tools like the 
Django Debug Toolbar and Yet Another Django Profiler. With my colleagues 
present, I've isolated and fixed significant speed problems inside of a few 
hours. I don't believe the Django ORM is inherently bad, although I do 
think that coders who use it should Know What They're Doing. Finally, I've 
referenced blog entries that talk about how Instagram and Disqus use Django 
on the backend for backend-y tasks.

Despite my best efforts, my colleagues are still pushing to have us rewrite 
large portions of our infrastructure as separate services before we try to 
fix them. For example, we have one slow REST endpoint that returns a 
boatload of user data, and so there's talk about using a new microservice 
for users in lieu of our existing Django models. Even if we are able to fix 
bottlenecks we encounter in a timely fashion, my colleagues fear that 
Django won't scale with the business.

I'm writing this post to garner additional evidence that Django will scale. 
Anything compelling (and preferably not obvious) that would help shed some 
light on Django's ability to scale would be *greatly* appreciated, as it's 
very difficult for me to defend my position that Django is a viable 
long-term solution without solid evidence to back up my claims. It 
certainly doesn't help that I don't have any experience scaling Django 
myself!

Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/83968c41-d415-4189-b33b-9f99b10b1c41%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Scaling Django

2016-02-03 Thread Avraham Serour
what do you mean by slow? can you measure in ms?

On Wed, Feb 3, 2016 at 5:30 PM, Joshua Pokotilow 
wrote:

> At the startup where I work, we've written a lot of our server code in
> Django. So far, we've adopted a "build it fast" mentality, so we invested
> very little time in optimizing our code. A small amount of load testing has
> revealed our codebase / infrastructure as it stands today needs to run
> faster and support more users.
>
> We recently hired some new engineers who are extremely skeptical that we
> should optimize our existing code. Their main concerns are:
>
> - We need to move to a service-oriented infrastructure because Django is
> too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
> - It's too easy to write slow queries using the Django ORM
> - It's hard to hire Django engineers
> - While Instagram and DISQUS use Django to service large numbers of
> people, they don't use it for any serious backend work
>
> After having worked with Django for the last 3 years, I'm a big believer
> in it, and I believe it would scale. To defend my position, I've pointed
> out to my colleagues that it's easy to identify bottlenecks with tools like
> the Django Debug Toolbar and Yet Another Django Profiler. With my
> colleagues present, I've isolated and fixed significant speed problems
> inside of a few hours. I don't believe the Django ORM is inherently bad,
> although I do think that coders who use it should Know What They're Doing.
> Finally, I've referenced blog entries that talk about how Instagram and
> Disqus use Django on the backend for backend-y tasks.
>
> Despite my best efforts, my colleagues are still pushing to have us
> rewrite large portions of our infrastructure as separate services before we
> try to fix them. For example, we have one slow REST endpoint that returns a
> boatload of user data, and so there's talk about using a new microservice
> for users in lieu of our existing Django models. Even if we are able to fix
> bottlenecks we encounter in a timely fashion, my colleagues fear that
> Django won't scale with the business.
>
> I'm writing this post to garner additional evidence that Django will
> scale. Anything compelling (and preferably not obvious) that would help
> shed some light on Django's ability to scale would be *greatly*
> appreciated, as it's very difficult for me to defend my position that
> Django is a viable long-term solution without solid evidence to back up my
> claims. It certainly doesn't help that I don't have any experience scaling
> Django myself!
>
> Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/83968c41-d415-4189-b33b-9f99b10b1c41%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAFWa6t%2BBkFpyKAQy8excUssKYpao%2BfGU8wJPVcQFT517X833vQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Scaling Django

2016-02-03 Thread Rafael E. Ferrero
Maybe I don't understand you very well, and for shure you have a very
specific problem to solve... but... do you read something of this?

http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views
https://www.digitalocean.com/community/tutorials/how-to-scale-django-beyond-the-basics
https://highperformancedjango.com/
http://talks.caktusgroup.com/djangocon/2013/scaling/#slide17
https://docs.djangoproject.com/en/1.8/faq/general/#does-django-scale


Rafael E. Ferrero

2016-02-03 12:30 GMT-03:00 Joshua Pokotilow :

> At the startup where I work, we've written a lot of our server code in
> Django. So far, we've adopted a "build it fast" mentality, so we invested
> very little time in optimizing our code. A small amount of load testing has
> revealed our codebase / infrastructure as it stands today needs to run
> faster and support more users.
>
> We recently hired some new engineers who are extremely skeptical that we
> should optimize our existing code. Their main concerns are:
>
> - We need to move to a service-oriented infrastructure because Django is
> too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
> - It's too easy to write slow queries using the Django ORM
> - It's hard to hire Django engineers
> - While Instagram and DISQUS use Django to service large numbers of
> people, they don't use it for any serious backend work
>
> After having worked with Django for the last 3 years, I'm a big believer
> in it, and I believe it would scale. To defend my position, I've pointed
> out to my colleagues that it's easy to identify bottlenecks with tools like
> the Django Debug Toolbar and Yet Another Django Profiler. With my
> colleagues present, I've isolated and fixed significant speed problems
> inside of a few hours. I don't believe the Django ORM is inherently bad,
> although I do think that coders who use it should Know What They're Doing.
> Finally, I've referenced blog entries that talk about how Instagram and
> Disqus use Django on the backend for backend-y tasks.
>
> Despite my best efforts, my colleagues are still pushing to have us
> rewrite large portions of our infrastructure as separate services before we
> try to fix them. For example, we have one slow REST endpoint that returns a
> boatload of user data, and so there's talk about using a new microservice
> for users in lieu of our existing Django models. Even if we are able to fix
> bottlenecks we encounter in a timely fashion, my colleagues fear that
> Django won't scale with the business.
>
> I'm writing this post to garner additional evidence that Django will
> scale. Anything compelling (and preferably not obvious) that would help
> shed some light on Django's ability to scale would be *greatly*
> appreciated, as it's very difficult for me to defend my position that
> Django is a viable long-term solution without solid evidence to back up my
> claims. It certainly doesn't help that I don't have any experience scaling
> Django myself!
>
> Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/83968c41-d415-4189-b33b-9f99b10b1c41%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAJJc_8Un4amXafG61xzt%2BbwY3bu4jOQ5LY-5MsxcXRMVm6Apvw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Scaling Django

2016-02-03 Thread Bill Blanchard
Let's try to adress some of their concerns:

- We need to move to a service-oriented infrastructure because Django is
> too monolithic


It depends on what your application does and what you're planning to do
with it in the future.  People are quick prescribe SOA as the end all way
to scale, but they tend to ignore the added complexity that comes with
building out and integrating smaller services.

 - It's too easy to write slow queries using the Django ORM


It's just as easy (arguably easier) to write slow queries using pure SQL or
any other ORM.  The ORM makes a lot of good decisions for mediocre
programmers (I'd put myself in that category).  If you're a great
programmer and have great programmers who really understand SQL, then
you're just as likely to get your ORM queries right as you are straight
SQL.

>
> - It's hard to hire Django engineers


Compared to what?  .NET or Java engineers?  Probably.  Harder than the
newest shiny javascript framework engineers? Probably not.  Django has
about as robust an  engineering population as Ruby/Rails does.  I don't
know what you'd convert to in order to make hiring easier.  All engineers
(especially good ones) are really hard to come by these days.  If you're
looking at outsourcing to Southwest Asia, then yes, the Django population
isn't as high as .NET/Java/PHP.  However, hiring challenges are most
typically defined by your location and your ability and/or willingness to
explore remote workers.

While Instagram and DISQUS use Django to service large numbers of people,
> they don't use it for any serious backend work


Reddit is also a large Django user.  All engineering decisions should be
made around what your particular needs are and what skills your team
possesses or is able to acquire.  Needs of an organization evolve over time
and the organizations adjust as they need to.

Many organizations start with a Python/Django or Ruby/Rails application to
build a product *quickly *which is what those stacks excel at.   A mantra
typically heard in the community is "don't optimize prematurely".  If
you're saying "man, we're going to hit a wall at 100,000 users", well you
need to get to 90,000 users first before worrying about 100,000.  Getting
the 90k users is the real hard part.

All this being said, your colleagues could be right to want to move off
Django.  We don't know much about your particular circumstances.

For more information on optimizing Django for scale, check out this book.
https://highperformancedjango.com/

Best of luck.

Bill




On Wed, Feb 3, 2016 at 10:30 AM, Joshua Pokotilow 
wrote:

> At the startup where I work, we've written a lot of our server code in
> Django. So far, we've adopted a "build it fast" mentality, so we invested
> very little time in optimizing our code. A small amount of load testing has
> revealed our codebase / infrastructure as it stands today needs to run
> faster and support more users.
>
> We recently hired some new engineers who are extremely skeptical that we
> should optimize our existing code. Their main concerns are:
>
> - We need to move to a service-oriented infrastructure because Django is
> too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
> - It's too easy to write slow queries using the Django ORM
> - It's hard to hire Django engineers
> - While Instagram and DISQUS use Django to service large numbers of
> people, they don't use it for any serious backend work
>
> After having worked with Django for the last 3 years, I'm a big believer
> in it, and I believe it would scale. To defend my position, I've pointed
> out to my colleagues that it's easy to identify bottlenecks with tools like
> the Django Debug Toolbar and Yet Another Django Profiler. With my
> colleagues present, I've isolated and fixed significant speed problems
> inside of a few hours. I don't believe the Django ORM is inherently bad,
> although I do think that coders who use it should Know What They're Doing.
> Finally, I've referenced blog entries that talk about how Instagram and
> Disqus use Django on the backend for backend-y tasks.
>
> Despite my best efforts, my colleagues are still pushing to have us
> rewrite large portions of our infrastructure as separate services before we
> try to fix them. For example, we have one slow REST endpoint that returns a
> boatload of user data, and so there's talk about using a new microservice
> for users in lieu of our existing Django models. Even if we are able to fix
> bottlenecks we encounter in a timely fashion, my colleagues fear that
> Django won't scale with the business.
>
> I'm writing this post to garner additional evidence that Django will
> scale. Anything compelling (and preferably not obvious) that would help
> shed some light on Django's ability to scale would be *greatly*
> appreciated, as it's very difficult for me to defend my position that
> Django is a viable long-term solution without solid evidence to back up my
> claims. It certainly doesn't help that I

Re: Scaling Django

2016-02-03 Thread Sergiy Khohlov
Hello,
 Your first  words have a answer. Swift coding always produces performance
problem.  This is expected.  Looks like  few new engineers  use another one
technology  and would not like to use django.  This a reason of his
criticism. Mostly low performance is related to the DB  performance. I'm
preferring avoid using ManyToMany  ability by due to high res usage at the
DB level. Writing  correct models and DB function helps in  the most case.
I have no idea about proposed solution but I definitely sure that code
produces bottleneck not programming language.  RubyOnRails has a same
problem such a Django for example.. Have you got  good performance at
JAva+Tomkat+Apache ?  I'm ready to see this high performance  ASP.
Half year ago I've rewritten GTS  service from C# to python. As result CPU
usage dropped  from 45% to 7-9% and memory usage from 1.5Gb  to 300kb.
Wrong solution, mistakes at the building project requirement stage
produces  more problem that selecting programming language.

Many thanks,

Serge


+380 636150445
skype: skhohlov

On Wed, Feb 3, 2016 at 5:30 PM, Joshua Pokotilow 
wrote:

> At the startup where I work, we've written a lot of our server code in
> Django. So far, we've adopted a "build it fast" mentality, so we invested
> very little time in optimizing our code. A small amount of load testing has
> revealed our codebase / infrastructure as it stands today needs to run
> faster and support more users.
>
> We recently hired some new engineers who are extremely skeptical that we
> should optimize our existing code. Their main concerns are:
>
> - We need to move to a service-oriented infrastructure because Django is
> too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
> - It's too easy to write slow queries using the Django ORM
> - It's hard to hire Django engineers
> - While Instagram and DISQUS use Django to service large numbers of
> people, they don't use it for any serious backend work
>
> After having worked with Django for the last 3 years, I'm a big believer
> in it, and I believe it would scale. To defend my position, I've pointed
> out to my colleagues that it's easy to identify bottlenecks with tools like
> the Django Debug Toolbar and Yet Another Django Profiler. With my
> colleagues present, I've isolated and fixed significant speed problems
> inside of a few hours. I don't believe the Django ORM is inherently bad,
> although I do think that coders who use it should Know What They're Doing.
> Finally, I've referenced blog entries that talk about how Instagram and
> Disqus use Django on the backend for backend-y tasks.
>
> Despite my best efforts, my colleagues are still pushing to have us
> rewrite large portions of our infrastructure as separate services before we
> try to fix them. For example, we have one slow REST endpoint that returns a
> boatload of user data, and so there's talk about using a new microservice
> for users in lieu of our existing Django models. Even if we are able to fix
> bottlenecks we encounter in a timely fashion, my colleagues fear that
> Django won't scale with the business.
>
> I'm writing this post to garner additional evidence that Django will
> scale. Anything compelling (and preferably not obvious) that would help
> shed some light on Django's ability to scale would be *greatly*
> appreciated, as it's very difficult for me to defend my position that
> Django is a viable long-term solution without solid evidence to back up my
> claims. It certainly doesn't help that I don't have any experience scaling
> Django myself!
>
> Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/83968c41-d415-4189-b33b-9f99b10b1c41%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CADTRxJPCNO2r%3D2-Vog0DGiXjmQof%3DQgH_RNjgySX1ENOkwUOrw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Scaling Django

2016-02-03 Thread Joshua Pokotilow
The service uses the Django REST Framework and takes multiple seconds to 
return a response. The response is a JSON array with thousands of 
dictionaries. We haven't yet investigated why it's slow, nor have we tried 
to cache / memoize anything to speed it up.

On Wednesday, February 3, 2016 at 10:46:25 AM UTC-5, Avraham Serour wrote:
>
> what do you mean by slow? can you measure in ms?
>
> On Wed, Feb 3, 2016 at 5:30 PM, Joshua Pokotilow  > wrote:
>
>> At the startup where I work, we've written a lot of our server code in 
>> Django. So far, we've adopted a "build it fast" mentality, so we invested 
>> very little time in optimizing our code. A small amount of load testing has 
>> revealed our codebase / infrastructure as it stands today needs to run 
>> faster and support more users.
>>
>> We recently hired some new engineers who are extremely skeptical that we 
>> should optimize our existing code. Their main concerns are:
>>
>> - We need to move to a service-oriented infrastructure because Django is 
>> too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
>> - It's too easy to write slow queries using the Django ORM
>> - It's hard to hire Django engineers
>> - While Instagram and DISQUS use Django to service large numbers of 
>> people, they don't use it for any serious backend work
>>
>> After having worked with Django for the last 3 years, I'm a big believer 
>> in it, and I believe it would scale. To defend my position, I've pointed 
>> out to my colleagues that it's easy to identify bottlenecks with tools like 
>> the Django Debug Toolbar and Yet Another Django Profiler. With my 
>> colleagues present, I've isolated and fixed significant speed problems 
>> inside of a few hours. I don't believe the Django ORM is inherently bad, 
>> although I do think that coders who use it should Know What They're Doing. 
>> Finally, I've referenced blog entries that talk about how Instagram and 
>> Disqus use Django on the backend for backend-y tasks.
>>
>> Despite my best efforts, my colleagues are still pushing to have us 
>> rewrite large portions of our infrastructure as separate services before we 
>> try to fix them. For example, we have one slow REST endpoint that returns a 
>> boatload of user data, and so there's talk about using a new microservice 
>> for users in lieu of our existing Django models. Even if we are able to fix 
>> bottlenecks we encounter in a timely fashion, my colleagues fear that 
>> Django won't scale with the business.
>>
>> I'm writing this post to garner additional evidence that Django will 
>> scale. Anything compelling (and preferably not obvious) that would help 
>> shed some light on Django's ability to scale would be *greatly* 
>> appreciated, as it's very difficult for me to defend my position that 
>> Django is a viable long-term solution without solid evidence to back up my 
>> claims. It certainly doesn't help that I don't have any experience scaling 
>> Django myself!
>>
>> Thank you.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com .
>> To post to this group, send email to django...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/83968c41-d415-4189-b33b-9f99b10b1c41%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/38b96b56-064a-4933-abb5-e6085cd24425%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


WHERE EXISTS / WHERE NOT EXISTS clause without using QuerySet.extra

2016-02-03 Thread john . parton
Greetings!

I'm trying to refactor a query to avoid using QuerySet.extra (as per the 
recommendation here: 
https://docs.djangoproject.com/en/1.9/ref/models/querysets/#extra)

Here's a simplified version of my code:

# testapp.models

class Parent(models.Model):
pass

class Child(models.Model):
parent = models.ForeignKey('testapp.Parent')

This is the query I am attempting to replace:

Parent.objects.extra(where=["""
NOT EXISTS (SELECT 1 FROM testapp_child WHERE testapp_child.parent_id = 
testapp_parent.id)
"""])

This is extremely similar to, but not the same as

Parent.objects.exclude(id__in=Child.objects.all().values('parent_id'))

Both queries should return the same rows, but the biggest difference is 
that PostgreSQL's query planner produces completely different plans. The 
performance difference can be huge, even for a relatively modest data set. 
(I believe my data set has something like 270k "parent" instances and 80k 
"child" instances.)

I tried searching this group as well as the ticket system, and couldn't 
find a solution to this exact problem (other than using the alternative 
query).

Thanks for taking the time to read through all of this! I am more than 
happy to open a ticket, but I thought I should post here first. 


More Technical Stuff

Here's the output of EXPLAIN ANALYZE on my actual models/data. I had to 
apply a LIMIT 100 because if I try to return the entire result, the second 
one completely hangs my computer. I bolded some relevant bits

Limit  (cost=0.71..9.70 rows=100 width=111) (actual time=0.074..0.483 
rows=100 loops=1)
  ->  *Merge Anti Join*  (cost=0.71..22435.69 rows=249613 width=111) 
(actual time=0.072..0.465 rows=100 loops=1)
Merge Cond: (catalogue_product.id = 
catalogue_productcategory.product_id)"
->  *Index Scan* using catalogue_product_pkey on catalogue_product  
(cost=0.42..16986.70 rows=292339 width=111) (actual time=0.020..0.285 
rows=150 loops=1)
->  *Index Only Scan* using catalogue_productcategory_9bea82de on 
catalogue_productcategory  (cost=0.29..3861.27 rows=81671 width=4) (actual 
time=0.037..0.070 rows=101 loops=1)
  Heap Fetches: 0
Total runtime: 

*0.568 ms*Here's the plan for the second query:
Limit  (cost=0.00..234229.95 rows=100 width=111) (actual 
time=31.087..1165.022 rows=100 loops=1)
  ->  *Seq Scan* on catalogue_product  (cost=0.00..342373919.34 rows=146170 
width=111) (actual time=31.087..1164.992 rows=100 loops=1)
Filter: (NOT (SubPlan 1))
Rows Removed by Filter: 5
SubPlan 1
  ->  *Materialize*  (cost=0.00..2138.07 rows=81671 width=4) 
(actual time=0.001..5.539 rows=80818 loops=105)
->  *Seq Scan* on catalogue_productcategory u0  
(cost=0.00..1409.71 rows=81671 width=4) (actual time=0.005..10.574 
rows=81671 loops=1)
Total runtime: 
*1165.255 ms*
As you can see the former is about ~2000 times faster than the latter.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/efc7a911-1ec7-4dc6-9b5a-c31832c071f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Scaling Django

2016-02-03 Thread Remco Gerlich
There is a book (ebook) named "High Performance Django" that has many
useful tips.

Also, new software developers are _always_ skeptical when they start a new
job. They have an old way of doing things from their previous job, "that's
not how we work here!" reflexes also from that old job, and they take a
while to adjust to the new tools.

 First let them get productive with what exists, _then_ let them gather
real statistics about real problems when they want to change something.

You can have scalibility issues and solve them with whatever framework,
Django included.

Remco Gerlich


On Wed, Feb 3, 2016 at 4:30 PM, Joshua Pokotilow 
wrote:

> At the startup where I work, we've written a lot of our server code in
> Django. So far, we've adopted a "build it fast" mentality, so we invested
> very little time in optimizing our code. A small amount of load testing has
> revealed our codebase / infrastructure as it stands today needs to run
> faster and support more users.
>
> We recently hired some new engineers who are extremely skeptical that we
> should optimize our existing code. Their main concerns are:
>
> - We need to move to a service-oriented infrastructure because Django is
> too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
> - It's too easy to write slow queries using the Django ORM
> - It's hard to hire Django engineers
> - While Instagram and DISQUS use Django to service large numbers of
> people, they don't use it for any serious backend work
>
> After having worked with Django for the last 3 years, I'm a big believer
> in it, and I believe it would scale. To defend my position, I've pointed
> out to my colleagues that it's easy to identify bottlenecks with tools like
> the Django Debug Toolbar and Yet Another Django Profiler. With my
> colleagues present, I've isolated and fixed significant speed problems
> inside of a few hours. I don't believe the Django ORM is inherently bad,
> although I do think that coders who use it should Know What They're Doing.
> Finally, I've referenced blog entries that talk about how Instagram and
> Disqus use Django on the backend for backend-y tasks.
>
> Despite my best efforts, my colleagues are still pushing to have us
> rewrite large portions of our infrastructure as separate services before we
> try to fix them. For example, we have one slow REST endpoint that returns a
> boatload of user data, and so there's talk about using a new microservice
> for users in lieu of our existing Django models. Even if we are able to fix
> bottlenecks we encounter in a timely fashion, my colleagues fear that
> Django won't scale with the business.
>
> I'm writing this post to garner additional evidence that Django will
> scale. Anything compelling (and preferably not obvious) that would help
> shed some light on Django's ability to scale would be *greatly*
> appreciated, as it's very difficult for me to defend my position that
> Django is a viable long-term solution without solid evidence to back up my
> claims. It certainly doesn't help that I don't have any experience scaling
> Django myself!
>
> Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/83968c41-d415-4189-b33b-9f99b10b1c41%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAFAGLK3wfsrqeDLcHX%2B3iaLVGoJb5ypXjoUdKKcjpm_UYakj1A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: WHERE EXISTS / WHERE NOT EXISTS clause without using QuerySet.extra

2016-02-03 Thread John P.
Sorry for the double-post, but I just searched the Django Developers group, 
and found this: "Using EXISTS instead of IN for subqueries" 
https://groups.google.com/forum/#!searchin/django-developers/WHERE$20EXISTS/django-developers/OG1unUV-MOU/VzkepyuaUDUJ
and this: "Support server-side cursors for queryset iteration in database 
backends" https://code.djangoproject.com/ticket/16614

I don't think this answers my specific question, because mine is more about 
trying to explicitly invoke the WHERE EXISTS clause instead of 
automatically replacing IN with WHERE EXISTS, but I thought I should put 
them here for anyone interested.

Thanks.

John

On Wednesday, February 3, 2016 at 10:03:31 AM UTC-6, John P. wrote:
>
> Greetings!
>
> I'm trying to refactor a query to avoid using QuerySet.extra (as per the 
> recommendation here: 
> https://docs.djangoproject.com/en/1.9/ref/models/querysets/#extra)
>
> Here's a simplified version of my code:
>
> # testapp.models
>
> class Parent(models.Model):
> pass
>
> class Child(models.Model):
> parent = models.ForeignKey('testapp.Parent')
>
> This is the query I am attempting to replace:
>
> Parent.objects.extra(where=["""
> NOT EXISTS (SELECT 1 FROM testapp_child WHERE testapp_child.parent_id 
> = testapp_parent.id)
> """])
>
> This is extremely similar to, but not the same as
>
> Parent.objects.exclude(id__in=Child.objects.all().values('parent_id'))
>
> Both queries should return the same rows, but the biggest difference is 
> that PostgreSQL's query planner produces completely different plans. The 
> performance difference can be huge, even for a relatively modest data set. 
> (I believe my data set has something like 270k "parent" instances and 80k 
> "child" instances.)
>
> I tried searching this group as well as the ticket system, and couldn't 
> find a solution to this exact problem (other than using the alternative 
> query).
>
> Thanks for taking the time to read through all of this! I am more than 
> happy to open a ticket, but I thought I should post here first. 
>
>
> More Technical Stuff
>
> Here's the output of EXPLAIN ANALYZE on my actual models/data. I had to 
> apply a LIMIT 100 because if I try to return the entire result, the second 
> one completely hangs my computer. I bolded some relevant bits
>
> Limit  (cost=0.71..9.70 rows=100 width=111) (actual time=0.074..0.483 
> rows=100 loops=1)
>   ->  *Merge Anti Join*  (cost=0.71..22435.69 rows=249613 width=111) 
> (actual time=0.072..0.465 rows=100 loops=1)
> Merge Cond: (catalogue_product.id = 
> catalogue_productcategory.product_id)"
> ->  *Index Scan* using catalogue_product_pkey on 
> catalogue_product  (cost=0.42..16986.70 rows=292339 width=111) (actual 
> time=0.020..0.285 rows=150 loops=1)
> ->  *Index Only Scan* using catalogue_productcategory_9bea82de on 
> catalogue_productcategory  (cost=0.29..3861.27 rows=81671 width=4) (actual 
> time=0.037..0.070 rows=101 loops=1)
>   Heap Fetches: 0
> Total runtime: 
>
> *0.568 ms*Here's the plan for the second query:
> Limit  (cost=0.00..234229.95 rows=100 width=111) (actual 
> time=31.087..1165.022 rows=100 loops=1)
>   ->  *Seq Scan* on catalogue_product  (cost=0.00..342373919.34 
> rows=146170 width=111) (actual time=31.087..1164.992 rows=100 loops=1)
> Filter: (NOT (SubPlan 1))
> Rows Removed by Filter: 5
> SubPlan 1
>   ->  *Materialize*  (cost=0.00..2138.07 rows=81671 width=4) 
> (actual time=0.001..5.539 rows=80818 loops=105)
> ->  *Seq Scan* on catalogue_productcategory u0  
> (cost=0.00..1409.71 rows=81671 width=4) (actual time=0.005..10.574 
> rows=81671 loops=1)
> Total runtime: 
> *1165.255 ms*
> As you can see the former is about ~2000 times faster than the latter.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/8301ed07-7c3a-451d-a6df-133317dce5be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Scaling Django

2016-02-03 Thread Avraham Serour
if your problem is the DB or network or small processor it won't help
rewriting the application.
The first step is to investigate the problem, then you can have solution,
sometimes people have a solution and then look for a problem, in your case
they want to leave python and django and are looking for problems with it.

more than which language, library and tech stack you use the system design
impacts a lot in the overall speed among other things, I've worked with
systems with too many indirections for example, the performance was not
impacted so much by the libraries we were using or not.

sometimes rewriting can be a good because of this, it is a way to give you
a chance to make a new design.

Avraham

On Wed, Feb 3, 2016 at 6:02 PM, Joshua Pokotilow 
wrote:

> The service uses the Django REST Framework and takes multiple seconds to
> return a response. The response is a JSON array with thousands of
> dictionaries. We haven't yet investigated why it's slow, nor have we tried
> to cache / memoize anything to speed it up.
>
> On Wednesday, February 3, 2016 at 10:46:25 AM UTC-5, Avraham Serour wrote:
>>
>> what do you mean by slow? can you measure in ms?
>>
>> On Wed, Feb 3, 2016 at 5:30 PM, Joshua Pokotilow 
>> wrote:
>>
>>> At the startup where I work, we've written a lot of our server code in
>>> Django. So far, we've adopted a "build it fast" mentality, so we invested
>>> very little time in optimizing our code. A small amount of load testing has
>>> revealed our codebase / infrastructure as it stands today needs to run
>>> faster and support more users.
>>>
>>> We recently hired some new engineers who are extremely skeptical that we
>>> should optimize our existing code. Their main concerns are:
>>>
>>> - We need to move to a service-oriented infrastructure because Django is
>>> too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
>>> - It's too easy to write slow queries using the Django ORM
>>> - It's hard to hire Django engineers
>>> - While Instagram and DISQUS use Django to service large numbers of
>>> people, they don't use it for any serious backend work
>>>
>>> After having worked with Django for the last 3 years, I'm a big believer
>>> in it, and I believe it would scale. To defend my position, I've pointed
>>> out to my colleagues that it's easy to identify bottlenecks with tools like
>>> the Django Debug Toolbar and Yet Another Django Profiler. With my
>>> colleagues present, I've isolated and fixed significant speed problems
>>> inside of a few hours. I don't believe the Django ORM is inherently bad,
>>> although I do think that coders who use it should Know What They're Doing.
>>> Finally, I've referenced blog entries that talk about how Instagram and
>>> Disqus use Django on the backend for backend-y tasks.
>>>
>>> Despite my best efforts, my colleagues are still pushing to have us
>>> rewrite large portions of our infrastructure as separate services before we
>>> try to fix them. For example, we have one slow REST endpoint that returns a
>>> boatload of user data, and so there's talk about using a new microservice
>>> for users in lieu of our existing Django models. Even if we are able to fix
>>> bottlenecks we encounter in a timely fashion, my colleagues fear that
>>> Django won't scale with the business.
>>>
>>> I'm writing this post to garner additional evidence that Django will
>>> scale. Anything compelling (and preferably not obvious) that would help
>>> shed some light on Django's ability to scale would be *greatly*
>>> appreciated, as it's very difficult for me to defend my position that
>>> Django is a viable long-term solution without solid evidence to back up my
>>> claims. It certainly doesn't help that I don't have any experience scaling
>>> Django myself!
>>>
>>> Thank you.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users...@googlegroups.com.
>>> To post to this group, send email to django...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-users.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-users/83968c41-d415-4189-b33b-9f99b10b1c41%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/

Re: Scaling Django

2016-02-03 Thread Larry Martell
I don't think there is a silver bullet that will fix all issues, nor
any one technology stack that will. I have a fairly good size django
app I built, and I did not consider performance all that much during
initial development. As the user base and dataset started to grow I
did see performance issues and I dealt with each one as they arose by
profiling the issue and seeing where the bottlenecks were. Often it
was not at all where I thought it would be. Each case had a different
solution, e.g.: database tuning, adding memory, writing custom queries
using temp tables, minimizing js code, optimizing jQuery code and DOM
manipulation, and so on.

On Wed, Feb 3, 2016 at 11:02 AM, Joshua Pokotilow  wrote:
> The service uses the Django REST Framework and takes multiple seconds to
> return a response. The response is a JSON array with thousands of
> dictionaries. We haven't yet investigated why it's slow, nor have we tried
> to cache / memoize anything to speed it up.
>
> On Wednesday, February 3, 2016 at 10:46:25 AM UTC-5, Avraham Serour wrote:
>>
>> what do you mean by slow? can you measure in ms?
>>
>> On Wed, Feb 3, 2016 at 5:30 PM, Joshua Pokotilow 
>> wrote:
>>>
>>> At the startup where I work, we've written a lot of our server code in
>>> Django. So far, we've adopted a "build it fast" mentality, so we invested
>>> very little time in optimizing our code. A small amount of load testing has
>>> revealed our codebase / infrastructure as it stands today needs to run
>>> faster and support more users.
>>>
>>> We recently hired some new engineers who are extremely skeptical that we
>>> should optimize our existing code. Their main concerns are:
>>>
>>> - We need to move to a service-oriented infrastructure because Django is
>>> too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
>>> - It's too easy to write slow queries using the Django ORM
>>> - It's hard to hire Django engineers
>>> - While Instagram and DISQUS use Django to service large numbers of
>>> people, they don't use it for any serious backend work
>>>
>>> After having worked with Django for the last 3 years, I'm a big believer
>>> in it, and I believe it would scale. To defend my position, I've pointed out
>>> to my colleagues that it's easy to identify bottlenecks with tools like the
>>> Django Debug Toolbar and Yet Another Django Profiler. With my colleagues
>>> present, I've isolated and fixed significant speed problems inside of a few
>>> hours. I don't believe the Django ORM is inherently bad, although I do think
>>> that coders who use it should Know What They're Doing. Finally, I've
>>> referenced blog entries that talk about how Instagram and Disqus use Django
>>> on the backend for backend-y tasks.
>>>
>>> Despite my best efforts, my colleagues are still pushing to have us
>>> rewrite large portions of our infrastructure as separate services before we
>>> try to fix them. For example, we have one slow REST endpoint that returns a
>>> boatload of user data, and so there's talk about using a new microservice
>>> for users in lieu of our existing Django models. Even if we are able to fix
>>> bottlenecks we encounter in a timely fashion, my colleagues fear that Django
>>> won't scale with the business.
>>>
>>> I'm writing this post to garner additional evidence that Django will
>>> scale. Anything compelling (and preferably not obvious) that would help shed
>>> some light on Django's ability to scale would be *greatly* appreciated, as
>>> it's very difficult for me to defend my position that Django is a viable
>>> long-term solution without solid evidence to back up my claims. It certainly
>>> doesn't help that I don't have any experience scaling Django myself!
>>>
>>> Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CACwCsY7p%2BfHjzUqoRhJn1Cy_djKpgGJRtL1TD210Gx6V9g%3DTMg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Scaling Django

2016-02-03 Thread Joshua Pokotilow
Thanks Bill. This was helpful. I understand that it's difficult to offer 
advice without too many specifics, so I'm hoping to get some high-level 
advice / examples of Django at scale, which you provided.

Thank you!

On Wednesday, February 3, 2016 at 10:49:51 AM UTC-5, Bill Blanchard wrote:
>
> Let's try to adress some of their concerns:
>
> - We need to move to a service-oriented infrastructure because Django is 
>> too monolithic 
>
>
> It depends on what your application does and what you're planning to do 
> with it in the future.  People are quick prescribe SOA as the end all way 
> to scale, but they tend to ignore the added complexity that comes with 
> building out and integrating smaller services.
>
>  - It's too easy to write slow queries using the Django ORM
>
>
> It's just as easy (arguably easier) to write slow queries using pure SQL 
> or any other ORM.  The ORM makes a lot of good decisions for mediocre 
> programmers (I'd put myself in that category).  If you're a great 
> programmer and have great programmers who really understand SQL, then 
> you're just as likely to get your ORM queries right as you are straight 
> SQL. 
>
>>
>> - It's hard to hire Django engineers
>
>
> Compared to what?  .NET or Java engineers?  Probably.  Harder than the 
> newest shiny javascript framework engineers? Probably not.  Django has 
> about as robust an  engineering population as Ruby/Rails does.  I don't 
> know what you'd convert to in order to make hiring easier.  All engineers 
> (especially good ones) are really hard to come by these days.  If you're 
> looking at outsourcing to Southwest Asia, then yes, the Django population 
> isn't as high as .NET/Java/PHP.  However, hiring challenges are most 
> typically defined by your location and your ability and/or willingness to 
> explore remote workers.
>
> While Instagram and DISQUS use Django to service large numbers of people, 
>> they don't use it for any serious backend work
>
>
> Reddit is also a large Django user.  All engineering decisions should be 
> made around what your particular needs are and what skills your team 
> possesses or is able to acquire.  Needs of an organization evolve over time 
> and the organizations adjust as they need to.
>
> Many organizations start with a Python/Django or Ruby/Rails application to 
> build a product *quickly *which is what those stacks excel at.   A mantra 
> typically heard in the community is "don't optimize prematurely".  If 
> you're saying "man, we're going to hit a wall at 100,000 users", well you 
> need to get to 90,000 users first before worrying about 100,000.  Getting 
> the 90k users is the real hard part.
>
> All this being said, your colleagues could be right to want to move off 
> Django.  We don't know much about your particular circumstances.
>
> For more information on optimizing Django for scale, check out this book.
> https://highperformancedjango.com/
>
> Best of luck.
>
> Bill
>
>
>
>
> On Wed, Feb 3, 2016 at 10:30 AM, Joshua Pokotilow  > wrote:
>
>> At the startup where I work, we've written a lot of our server code in 
>> Django. So far, we've adopted a "build it fast" mentality, so we invested 
>> very little time in optimizing our code. A small amount of load testing has 
>> revealed our codebase / infrastructure as it stands today needs to run 
>> faster and support more users.
>>
>> We recently hired some new engineers who are extremely skeptical that we 
>> should optimize our existing code. Their main concerns are:
>>
>> - We need to move to a service-oriented infrastructure because Django is 
>> too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
>> - It's too easy to write slow queries using the Django ORM
>> - It's hard to hire Django engineers
>> - While Instagram and DISQUS use Django to service large numbers of 
>> people, they don't use it for any serious backend work
>>
>> After having worked with Django for the last 3 years, I'm a big believer 
>> in it, and I believe it would scale. To defend my position, I've pointed 
>> out to my colleagues that it's easy to identify bottlenecks with tools like 
>> the Django Debug Toolbar and Yet Another Django Profiler. With my 
>> colleagues present, I've isolated and fixed significant speed problems 
>> inside of a few hours. I don't believe the Django ORM is inherently bad, 
>> although I do think that coders who use it should Know What They're Doing. 
>> Finally, I've referenced blog entries that talk about how Instagram and 
>> Disqus use Django on the backend for backend-y tasks.
>>
>> Despite my best efforts, my colleagues are still pushing to have us 
>> rewrite large portions of our infrastructure as separate services before we 
>> try to fix them. For example, we have one slow REST endpoint that returns a 
>> boatload of user data, and so there's talk about using a new microservice 
>> for users in lieu of our existing Django models. Even if we are able to fix 
>> bottlenecks we encounter in a 

Re: Scaling Django

2016-02-03 Thread Joshua Pokotilow
Thank you Sergiy! I agree that the code needs to be fixed.

We don't have a Tomcat endpoint to compare with, although I did scare my 
coworkers a bit when we profiled a Django endpoint that took 300 - 400ms to 
return a response due (ostensibly) in large part to form object 
instantiation. Specifically, the bottleneck seemed to be that forms 
deep-copy their fields on instantiation.

On Wednesday, February 3, 2016 at 11:01:43 AM UTC-5, Sergiy Khohlov wrote:
>
> Hello, 
>  Your first  words have a answer. Swift coding always produces performance 
> problem.  This is expected.  Looks like  few new engineers  use another one 
> technology  and would not like to use django.  This a reason of his 
> criticism. Mostly low performance is related to the DB  performance. I'm 
> preferring avoid using ManyToMany  ability by due to high res usage at the 
> DB level. Writing  correct models and DB function helps in  the most case. 
> I have no idea about proposed solution but I definitely sure that code 
> produces bottleneck not programming language.  RubyOnRails has a same 
> problem such a Django for example.. Have you got  good performance at 
> JAva+Tomkat+Apache ?  I'm ready to see this high performance  ASP. 
> Half year ago I've rewritten GTS  service from C# to python. As result CPU 
> usage dropped  from 45% to 7-9% and memory usage from 1.5Gb  to 300kb. 
> Wrong solution, mistakes at the building project requirement stage 
> produces  more problem that selecting programming language.
>
> Many thanks,
>
> Serge
>
>
> +380 636150445
> skype: skhohlov
>
> On Wed, Feb 3, 2016 at 5:30 PM, Joshua Pokotilow  > wrote:
>
>> At the startup where I work, we've written a lot of our server code in 
>> Django. So far, we've adopted a "build it fast" mentality, so we invested 
>> very little time in optimizing our code. A small amount of load testing has 
>> revealed our codebase / infrastructure as it stands today needs to run 
>> faster and support more users.
>>
>> We recently hired some new engineers who are extremely skeptical that we 
>> should optimize our existing code. Their main concerns are:
>>
>> - We need to move to a service-oriented infrastructure because Django is 
>> too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
>> - It's too easy to write slow queries using the Django ORM
>> - It's hard to hire Django engineers
>> - While Instagram and DISQUS use Django to service large numbers of 
>> people, they don't use it for any serious backend work
>>
>> After having worked with Django for the last 3 years, I'm a big believer 
>> in it, and I believe it would scale. To defend my position, I've pointed 
>> out to my colleagues that it's easy to identify bottlenecks with tools like 
>> the Django Debug Toolbar and Yet Another Django Profiler. With my 
>> colleagues present, I've isolated and fixed significant speed problems 
>> inside of a few hours. I don't believe the Django ORM is inherently bad, 
>> although I do think that coders who use it should Know What They're Doing. 
>> Finally, I've referenced blog entries that talk about how Instagram and 
>> Disqus use Django on the backend for backend-y tasks.
>>
>> Despite my best efforts, my colleagues are still pushing to have us 
>> rewrite large portions of our infrastructure as separate services before we 
>> try to fix them. For example, we have one slow REST endpoint that returns a 
>> boatload of user data, and so there's talk about using a new microservice 
>> for users in lieu of our existing Django models. Even if we are able to fix 
>> bottlenecks we encounter in a timely fashion, my colleagues fear that 
>> Django won't scale with the business.
>>
>> I'm writing this post to garner additional evidence that Django will 
>> scale. Anything compelling (and preferably not obvious) that would help 
>> shed some light on Django's ability to scale would be *greatly* 
>> appreciated, as it's very difficult for me to defend my position that 
>> Django is a viable long-term solution without solid evidence to back up my 
>> claims. It certainly doesn't help that I don't have any experience scaling 
>> Django myself!
>>
>> Thank you.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com .
>> To post to this group, send email to django...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/83968c41-d415-4189-b33b-9f99b10b1c41%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed t

Re: Scaling Django

2016-02-03 Thread Joshua Pokotilow
Thanks Remco. I'll look at High Performance Django.

On Wednesday, February 3, 2016 at 11:05:11 AM UTC-5, Remco Gerlich wrote:
>
> There is a book (ebook) named "High Performance Django" that has many 
> useful tips.
>
> Also, new software developers are _always_ skeptical when they start a new 
> job. They have an old way of doing things from their previous job, "that's 
> not how we work here!" reflexes also from that old job, and they take a 
> while to adjust to the new tools.
>
>  First let them get productive with what exists, _then_ let them gather 
> real statistics about real problems when they want to change something.
>
> You can have scalibility issues and solve them with whatever framework, 
> Django included.
>
> Remco Gerlich
>
>
> On Wed, Feb 3, 2016 at 4:30 PM, Joshua Pokotilow  > wrote:
>
>> At the startup where I work, we've written a lot of our server code in 
>> Django. So far, we've adopted a "build it fast" mentality, so we invested 
>> very little time in optimizing our code. A small amount of load testing has 
>> revealed our codebase / infrastructure as it stands today needs to run 
>> faster and support more users.
>>
>> We recently hired some new engineers who are extremely skeptical that we 
>> should optimize our existing code. Their main concerns are:
>>
>> - We need to move to a service-oriented infrastructure because Django is 
>> too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
>> - It's too easy to write slow queries using the Django ORM
>> - It's hard to hire Django engineers
>> - While Instagram and DISQUS use Django to service large numbers of 
>> people, they don't use it for any serious backend work
>>
>> After having worked with Django for the last 3 years, I'm a big believer 
>> in it, and I believe it would scale. To defend my position, I've pointed 
>> out to my colleagues that it's easy to identify bottlenecks with tools like 
>> the Django Debug Toolbar and Yet Another Django Profiler. With my 
>> colleagues present, I've isolated and fixed significant speed problems 
>> inside of a few hours. I don't believe the Django ORM is inherently bad, 
>> although I do think that coders who use it should Know What They're Doing. 
>> Finally, I've referenced blog entries that talk about how Instagram and 
>> Disqus use Django on the backend for backend-y tasks.
>>
>> Despite my best efforts, my colleagues are still pushing to have us 
>> rewrite large portions of our infrastructure as separate services before we 
>> try to fix them. For example, we have one slow REST endpoint that returns a 
>> boatload of user data, and so there's talk about using a new microservice 
>> for users in lieu of our existing Django models. Even if we are able to fix 
>> bottlenecks we encounter in a timely fashion, my colleagues fear that 
>> Django won't scale with the business.
>>
>> I'm writing this post to garner additional evidence that Django will 
>> scale. Anything compelling (and preferably not obvious) that would help 
>> shed some light on Django's ability to scale would be *greatly* 
>> appreciated, as it's very difficult for me to defend my position that 
>> Django is a viable long-term solution without solid evidence to back up my 
>> claims. It certainly doesn't help that I don't have any experience scaling 
>> Django myself!
>>
>> Thank you.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com .
>> To post to this group, send email to django...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/83968c41-d415-4189-b33b-9f99b10b1c41%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/eff1cd01-58f0-48d8-b58d-ef4d71a6ab29%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Scaling Django

2016-02-03 Thread Joshua Pokotilow
Thank you! I agree that we need to investigate before coming up with a 
solution.

On Wednesday, February 3, 2016 at 11:11:04 AM UTC-5, Avraham Serour wrote:
>
> if your problem is the DB or network or small processor it won't help 
> rewriting the application.
> The first step is to investigate the problem, then you can have solution, 
> sometimes people have a solution and then look for a problem, in your case 
> they want to leave python and django and are looking for problems with it.
>
> more than which language, library and tech stack you use the system design 
> impacts a lot in the overall speed among other things, I've worked with 
> systems with too many indirections for example, the performance was not 
> impacted so much by the libraries we were using or not.
>
> sometimes rewriting can be a good because of this, it is a way to give you 
> a chance to make a new design.
>
> Avraham
>
> On Wed, Feb 3, 2016 at 6:02 PM, Joshua Pokotilow  > wrote:
>
>> The service uses the Django REST Framework and takes multiple seconds to 
>> return a response. The response is a JSON array with thousands of 
>> dictionaries. We haven't yet investigated why it's slow, nor have we tried 
>> to cache / memoize anything to speed it up.
>>
>> On Wednesday, February 3, 2016 at 10:46:25 AM UTC-5, Avraham Serour wrote:
>>>
>>> what do you mean by slow? can you measure in ms?
>>>
>>> On Wed, Feb 3, 2016 at 5:30 PM, Joshua Pokotilow  
>>> wrote:
>>>
 At the startup where I work, we've written a lot of our server code in 
 Django. So far, we've adopted a "build it fast" mentality, so we invested 
 very little time in optimizing our code. A small amount of load testing 
 has 
 revealed our codebase / infrastructure as it stands today needs to run 
 faster and support more users.

 We recently hired some new engineers who are extremely skeptical that 
 we should optimize our existing code. Their main concerns are:

 - We need to move to a service-oriented infrastructure because Django 
 is too monolithic (monolithic = technology lock-in & difficult to 
 troubleshoot)
 - It's too easy to write slow queries using the Django ORM
 - It's hard to hire Django engineers
 - While Instagram and DISQUS use Django to service large numbers of 
 people, they don't use it for any serious backend work

 After having worked with Django for the last 3 years, I'm a big 
 believer in it, and I believe it would scale. To defend my position, I've 
 pointed out to my colleagues that it's easy to identify bottlenecks with 
 tools like the Django Debug Toolbar and Yet Another Django Profiler. With 
 my colleagues present, I've isolated and fixed significant speed problems 
 inside of a few hours. I don't believe the Django ORM is inherently bad, 
 although I do think that coders who use it should Know What They're Doing. 
 Finally, I've referenced blog entries that talk about how Instagram and 
 Disqus use Django on the backend for backend-y tasks.

 Despite my best efforts, my colleagues are still pushing to have us 
 rewrite large portions of our infrastructure as separate services before 
 we 
 try to fix them. For example, we have one slow REST endpoint that returns 
 a 
 boatload of user data, and so there's talk about using a new microservice 
 for users in lieu of our existing Django models. Even if we are able to 
 fix 
 bottlenecks we encounter in a timely fashion, my colleagues fear that 
 Django won't scale with the business.

 I'm writing this post to garner additional evidence that Django will 
 scale. Anything compelling (and preferably not obvious) that would help 
 shed some light on Django's ability to scale would be *greatly* 
 appreciated, as it's very difficult for me to defend my position that 
 Django is a viable long-term solution without solid evidence to back up my 
 claims. It certainly doesn't help that I don't have any experience scaling 
 Django myself!

 Thank you.

 -- 
 You received this message because you are subscribed to the Google 
 Groups "Django users" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to django-users...@googlegroups.com.
 To post to this group, send email to django...@googlegroups.com.
 Visit this group at https://groups.google.com/group/django-users.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/django-users/83968c41-d415-4189-b33b-9f99b10b1c41%40googlegroups.com
  
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" g

Is there a hook to run an SQL Query at every Postgres session start?

2016-02-03 Thread Bobby Mozumder
I'm looking to use PREPARE/EXECUTE statements, to eliminate my Query Planning 
time from every Web request.

This can be done via SQL PREPARE/EXECUTE statements.

But, Postgres only supports PREPARE statements on a connection 
session-by-session basis.  

To do this with Django, I need to be able to run an SQL Query that runs the 
PREPARE statement for every connection to the Database. I only need to do this 
once per DB connection, as I have enabled Persistent connections.  

The code to run, if I put it into the App.view, would be something like this:

class PreparedView(View):
def prepare_db():
cursor = connection.cursor()
cursor.execute(“”"
PREPARE prepared_query (int, text, etc) AS
  SELECT … some query … 
“”"
return cursor.close()

I’m looking through the Django code to see where I can run some sort of Query.  
Maybe I can subclass 
django.db.backends.postgresql.base.DatabaseWrapper.init_db_connection_state()?

Also, where can I hook that into my app?  If I subclass that, I’m not sure if I 
can/should put that in the Model, View, or URL of modules of my app, so that 
it’s called?  Do I create an entirely new version of 
django.db.backends.postgresql?

Ideally this would work with both the basic built-in development server 
connecting to a local host Postgres, as well as in a uWSGI environment with 
possible pgBouncer in the stack somewhere.

Thanks!

-bobby

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/E91526A2-C5C1-449F-8DC9-0A56CDB74738%40gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Is there a hook to run an SQL Query at every Postgres session start?

2016-02-03 Thread Simon Charette
Hi Bobby,

I'm not sure this is what you are looking for but it looks like 
`connection_created`[1] signal might do.

Cheers,
Simon

[1] https://docs.djangoproject.com/en/1.9/ref/signals/#connection-created

Le mercredi 3 février 2016 12:25:36 UTC-5, Bobby Mozumder a écrit :
>
> I'm looking to use PREPARE/EXECUTE statements, to eliminate my Query 
> Planning time from every Web request.
>
> This can be done via SQL PREPARE/EXECUTE statements.
>
> But, Postgres only supports PREPARE statements on a connection 
> session-by-session basis.  
>
> To do this with Django, I need to be able to run an SQL Query that runs 
> the PREPARE statement for every connection to the Database. I only need to 
> do this once per DB connection, as I have enabled Persistent connections.  
>
> The code to run, if I put it into the App.view, would be something like 
> this:
>
> class PreparedView(View):
> def prepare_db():
> cursor = connection.cursor()
> cursor.execute(“”"
> PREPARE prepared_query (int, text, etc) AS
>   SELECT … some query … 
> “”"
> return cursor.close()
>
> I’m looking through the Django code to see where I can run some sort of 
> Query.  Maybe I can subclass 
> django.db.backends.postgresql.base.DatabaseWrapper.init_db_connection_state()?
>
> Also, where can I hook that into my app?  If I subclass that, I’m not sure 
> if I can/should put that in the Model, View, or URL of modules of my app, 
> so that it’s called?  Do I create an entirely new version of 
> django.db.backends.postgresql?
>
> Ideally this would work with both the basic built-in development server 
> connecting to a local host Postgres, as well as in a uWSGI environment with 
> possible pgBouncer in the stack somewhere.
>
> Thanks!
>
> -bobby
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/06a0a777-9823-4112-9683-131e277d0c2b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Scaling Django

2016-02-03 Thread Fred Stluka

  
  
Joshua,

My team is producing a Django app with a small number of 
users, so we haven't worried too much about performance yet,
but we know we may have to some day, so I've accumulated a 
list of ways to improve performance in a JIRA ticket for if/when 
it becomes a priority.  

We've done some of them already, with a quick and easy 
thousand-fold increase in speed.

Here's a cut/paste of our ideas.  I hope you find it useful.



  Optimize the site for speed 
  
I/O speed
  
Prime candidate #3. May be costing us
at least a couple seconds.
  Easy fix
Especially to phones with slow Internet connections
Minimize the size of the HTML, JS, CSS and image files
  
Combine CSS into a single file, and JS into a single
  file
Minify the HTML, JS, and CSS
Compress (zip) the HTML, JS, and CSS
  
May want to use gulp to minify and compress.
  See:
  
http://revsys.com/blog/2014/oct/21/ultimate-front-end-development-setup/
  

  

Move CSS to the top and JS to the bottom of HTML
  files
  
See "How to improve web page performance from
  RevSys" below
  

http://www.revsys.com/12days/front-end-performance/
New HTML5 "Picture" element ("art direction"
  technique) and CSS calc() function to optimize image
  download speed:
  
http://arstechnica.com/information-technology/2014/09/how-a-new-html-element-will-make-the-web-faster/
https://longhandpixels.net/blog/2014/02/complete-guide-picture-element
https://developer.mozilla.org/en-US/docs/Web/CSS/calc
https://www.youtube.com/watch?v=QINlm3vjnaY
http://alistapart.com/article/responsive-images-in-practice
  

  

Limit the size of image files uploaded by the patients. 
  Scale them smaller as they are uploaded if necessary.
Optimize caching by browsers
  
Far future expires headers
  
See "How to improve web page performance from
  RevSys" below
http://www.revsys.com/12days/front-end-performance/
  

Version number embedded in resource names for force
  re-load at new release
  

  

  
  
CPU speed
  
Do as much as possible in JS on the local device, not in
  Python/Django on the shared server
  

  
  
Disk speed
  
Use the AWS Console to change the disk drives of the
  server from magnetic
  disks to SSDs.
Set up a RAID volume
  

  
  
DB speed
  
Do all selects/joins/filtering in the DB. Do not query
  lots of data
  from the DB and then iterate over it or otherwise filter
  it in Python.
  Especially, do not query lots of data from the DB, pass it
  all to the
  browser, and iterate over it or otherwise filter it in
  _javascript_.
Create DB indexes for all fields used frequently in
  SELECTs
  
But don't over-index
  

Move the DB server to a different Linux server than the
  Web server, to split the load between 2 servers, if the DB
  is the bottleneck. Use Amazon RDS
http://www.revsys.com/12days/finding-sources-of-slowness/
  
EXPLAIN PLAN
EXPLAIN ANALYZE
  

  

  
  
Django DB speed:
  
Monitoring DB access:
  
http://stackoverflow.com/questions/2133627/using-django-db-connection-queries
  

select_related
  
https://docs.djangoproject.com/en/1.7/ref/models/querysets/#select-related
  

prefetch_related
  
https://docs.djangoproject.com/en/1.7/ref/models/querysets/#django.db.models.query.QuerySet.prefetch_related
  

Optimize Django's ORM access to DB:
  

  

Re: Scaling Django

2016-02-03 Thread orzodk

While optimizing the code will bring you improvements and you shouldn't
stop doing this, for the most part (as noted from Rafael's resources)
you should update your architecture to support Django in scaling.

As you mentioned, instead of hitting the DB for every multi-second API
call you scale it by caching results so they aren't recalculated on demand.


Microservices are something you should work toward by refactoring rather
than rewriting as that is a really great way to kill a start up.

http://steveblank.com/2011/01/25/startup-suicide-%E2%80%93-rewriting-the-code/

"Rafael E. Ferrero"  writes:

> Maybe I don't understand you very well, and for shure you have a very specific
> problem to solve... but... do you read something of this?
>
> http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views
> https://www.digitalocean.com/community/tutorials/how-to-scale-django-beyond-the-basics
> https://highperformancedjango.com/
> http://talks.caktusgroup.com/djangocon/2013/scaling/#slide17
> https://docs.djangoproject.com/en/1.8/faq/general/#does-django-scale
>
> Rafael E. Ferrero
>
> 2016-02-03 12:30 GMT-03:00 Joshua Pokotilow :
>
> At the startup where I work, we've written a lot of our server code in
> Django. So far, we've adopted a "build it fast" mentality, so we invested
> very little time in optimizing our code. A small amount of load testing 
> has
> revealed our codebase / infrastructure as it stands today needs to run
> faster and support more users.
> 
> We recently hired some new engineers who are extremely skeptical that we
> should optimize our existing code. Their main concerns are:
> 
> - We need to move to a service-oriented infrastructure because Django is 
> too
> monolithic (monolithic = technology lock-in & difficult to troubleshoot)
> - It's too easy to write slow queries using the Django ORM
> - It's hard to hire Django engineers
> - While Instagram and DISQUS use Django to service large numbers of 
> people,
> they don't use it for any serious backend work
> 
> After having worked with Django for the last 3 years, I'm a big believer 
> in
> it, and I believe it would scale. To defend my position, I've pointed out 
> to
> my colleagues that it's easy to identify bottlenecks with tools like the
> Django Debug Toolbar and Yet Another Django Profiler. With my colleagues
> present, I've isolated and fixed significant speed problems inside of a 
> few
> hours. I don't believe the Django ORM is inherently bad, although I do 
> think
> that coders who use it should Know What They're Doing. Finally, I've
> referenced blog entries that talk about how Instagram and Disqus use 
> Django
> on the backend for backend-y tasks.
> 
> Despite my best efforts, my colleagues are still pushing to have us 
> rewrite
> large portions of our infrastructure as separate services before we try to
> fix them. For example, we have one slow REST endpoint that returns a
> boatload of user data, and so there's talk about using a new microservice
> for users in lieu of our existing Django models. Even if we are able to 
> fix
> bottlenecks we encounter in a timely fashion, my colleagues fear that 
> Django
> won't scale with the business.
> 
> 
> 
> I'm writing this post to garner additional evidence that Django will 
> scale.
> Anything compelling (and preferably not obvious) that would help shed some
> light on Django's ability to scale would be *greatly* appreciated, as it's
> very difficult for me to defend my position that Django is a viable
> long-term solution without solid evidence to back up my claims. It 
> certainly
> doesn't help that I don't have any experience scaling Django myself!
> 
> 
> Thank you.
>
> -- 
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> 
> https://groups.google.com/d/msgid/django-users/83968c41-d415-4189-b33b-9f99b10b1c41%40googlegroups.com
>.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/m2k2mlpvhe.fsf%40ip-192-168-1-5.us-west-

Re: Scaling Django

2016-02-03 Thread Daniel Chimeno
As you said the project is using DRF for an API, it came to my mind some 
blog post I've read about it:

   - 
   
http://ses4j.github.io/2015/11/23/optimizing-slow-django-rest-framework-performance/
   - 
   https://www.dabapps.com/blog/api-performance-profiling-django-rest-framework/
   - 
   https://docs.djangoproject.com/es/1.9/ref/models/querysets/#prefetch-related
   
I'm sure with some little tricks (that shouldn't be tricks after all) 
you'll go over that situation.
As others said, first look the problem, then search the solution.

In that specific case that you are getting thousand of results from 
database, you can go further in:
- SQL
- Caching
- Serialize
- Pagination

Hope it helps.


El miércoles, 3 de febrero de 2016, 16:30:05 (UTC+1), Joshua Pokotilow 
escribió:
>
> At the startup where I work, we've written a lot of our server code in 
> Django. So far, we've adopted a "build it fast" mentality, so we invested 
> very little time in optimizing our code. A small amount of load testing has 
> revealed our codebase / infrastructure as it stands today needs to run 
> faster and support more users.
>
> We recently hired some new engineers who are extremely skeptical that we 
> should optimize our existing code. Their main concerns are:
>
> - We need to move to a service-oriented infrastructure because Django is 
> too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
> - It's too easy to write slow queries using the Django ORM
> - It's hard to hire Django engineers
> - While Instagram and DISQUS use Django to service large numbers of 
> people, they don't use it for any serious backend work
>
> After having worked with Django for the last 3 years, I'm a big believer 
> in it, and I believe it would scale. To defend my position, I've pointed 
> out to my colleagues that it's easy to identify bottlenecks with tools like 
> the Django Debug Toolbar and Yet Another Django Profiler. With my 
> colleagues present, I've isolated and fixed significant speed problems 
> inside of a few hours. I don't believe the Django ORM is inherently bad, 
> although I do think that coders who use it should Know What They're Doing. 
> Finally, I've referenced blog entries that talk about how Instagram and 
> Disqus use Django on the backend for backend-y tasks.
>
> Despite my best efforts, my colleagues are still pushing to have us 
> rewrite large portions of our infrastructure as separate services before we 
> try to fix them. For example, we have one slow REST endpoint that returns a 
> boatload of user data, and so there's talk about using a new microservice 
> for users in lieu of our existing Django models. Even if we are able to fix 
> bottlenecks we encounter in a timely fashion, my colleagues fear that 
> Django won't scale with the business.
>
> I'm writing this post to garner additional evidence that Django will 
> scale. Anything compelling (and preferably not obvious) that would help 
> shed some light on Django's ability to scale would be *greatly* 
> appreciated, as it's very difficult for me to defend my position that 
> Django is a viable long-term solution without solid evidence to back up my 
> claims. It certainly doesn't help that I don't have any experience scaling 
> Django myself!
>
> Thank you.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b831b050-fb2f-4718-a9c6-610c6152865a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Scaling Django

2016-02-03 Thread Russell Keith-Magee
On Wed, Feb 3, 2016 at 11:30 PM, Joshua Pokotilow 
wrote:

> At the startup where I work, we've written a lot of our server code in
> Django. So far, we've adopted a "build it fast" mentality, so we invested
> very little time in optimizing our code. A small amount of load testing has
> revealed our codebase / infrastructure as it stands today needs to run
> faster and support more users.
>
> We recently hired some new engineers who are extremely skeptical that we
> should optimize our existing code. Their main concerns are:
>
> - We need to move to a service-oriented infrastructure because Django is
> too monolithic (monolithic = technology lock-in & difficult to troubleshoot)
> - It's too easy to write slow queries using the Django ORM
> - It's hard to hire Django engineers
> - While Instagram and DISQUS use Django to service large numbers of
> people, they don't use it for any serious backend work
>
> After having worked with Django for the last 3 years, I'm a big believer
> in it, and I believe it would scale. To defend my position, I've pointed
> out to my colleagues that it's easy to identify bottlenecks with tools like
> the Django Debug Toolbar and Yet Another Django Profiler. With my
> colleagues present, I've isolated and fixed significant speed problems
> inside of a few hours. I don't believe the Django ORM is inherently bad,
> although I do think that coders who use it should Know What They're Doing.
> Finally, I've referenced blog entries that talk about how Instagram and
> Disqus use Django on the backend for backend-y tasks.
>
> Despite my best efforts, my colleagues are still pushing to have us
> rewrite large portions of our infrastructure as separate services before we
> try to fix them. For example, we have one slow REST endpoint that returns a
> boatload of user data, and so there's talk about using a new microservice
> for users in lieu of our existing Django models. Even if we are able to fix
> bottlenecks we encounter in a timely fashion, my colleagues fear that
> Django won't scale with the business.
>

My immediate reaction, knowing nothing about the site or it’s codebase -

1) There’s nothing they’re proposing that excludes Django from the mix.
2) From an engineering management perspective, the solution they’re
proposing is much more concerning than the problems you’re describing.

My suggestion for convincing management:

Tell them that you can write Microservices in Django. Because you can.
Build a minimal Django stack - something that just returns a static Hello
World - and do some load testing. This will prove that Django can serve
high load - or at least as much load as whatever technology they’re
proposing.

Tell them that Microservices is just a new word for something software
engineers have been calling “High cohesion, low coupling” since the 1960s.
The only difference is that this time, instead of using the low latency,
high speed interface of a function call, we’re using the slow, unreliable
transfer of HTTP. If you’re actually focussing on performance, it’s trivial
to build a high cohesion, low coupling stack in *any* technology. All that
Webservices do is enforce this by making the inter-module barrier obvious.

Tell them that Microservices aren’t magic fairy sauce for speed. If the
issue with your existing codebase is the speed of database queries, that
problem isn’t going to go away by putting your code behind microservices.
You’re just going to add the cost of inter-service HTTP transfer to the
overhead of making a query. And if you’re putting something essential -
like the user database - behind a service, then you’d better be prepared to
add the round-trip time of a HTTP lookup onto Every. Single. Page. (Tell me
again how this is good for performance?)

Teach them about Second Systems Syndrome [1] [2].

[1]
https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effect
[2] http://coliveira.net/software/what-is-second-system-syndrome/

Tell them that while Django engineers might be hard to hire, they’re also
relatively easy to grow from scratch. DjangoGirls proves you can take
people with no experience in programming and make them competent Django
developers. Take someone with a history in *any* programming language, and
you can teach them Django; hire one or two Django experts to provide an
internal knowledge and review, and you’re set.

Lastly, tell them that despite their protestations, your site isn’t
Instagram, Disqus, or anything like it. 99% of web sites are not in the top
1% of websites by traffic. Your website is *not* in the top 1%. It might be
one day. But it isn’t now. And if you’re *ever* in a position where you
might end up in the top 1% - I can *guarantee* that it will be accompanied
by a metric buttload of engineers and money who will have a lot more
experience in scaling large scale services than any of the people who are
proposing microservices as a silver bullet.

Now - I’m saying all this without having actually seen your code or

Re: Is there a hook to run an SQL Query at every Postgres session start?

2016-02-03 Thread Bobby Mozumder
Thanks Simon this works great so far =^)

-bobby

> On Feb 3, 2016, at 12:34 PM, Simon Charette  wrote:
> 
> Hi Bobby,
> 
> I'm not sure this is what you are looking for but it looks like 
> `connection_created`[1] signal might do.
> 
> Cheers,
> Simon
> 
> [1] https://docs.djangoproject.com/en/1.9/ref/signals/#connection-created
> 
> Le mercredi 3 février 2016 12:25:36 UTC-5, Bobby Mozumder a écrit :
> I'm looking to use PREPARE/EXECUTE statements, to eliminate my Query Planning 
> time from every Web request.
> 
> This can be done via SQL PREPARE/EXECUTE statements.
> 
> But, Postgres only supports PREPARE statements on a connection 
> session-by-session basis.  
> 
> To do this with Django, I need to be able to run an SQL Query that runs the 
> PREPARE statement for every connection to the Database. I only need to do 
> this once per DB connection, as I have enabled Persistent connections.  
> 
> The code to run, if I put it into the App.view, would be something like this:
> 
> class PreparedView(View):
> def prepare_db():
> cursor = connection.cursor()
> cursor.execute(“”"
> PREPARE prepared_query (int, text, etc) AS
>   SELECT … some query … 
> “”"
> return cursor.close()
> 
> I’m looking through the Django code to see where I can run some sort of 
> Query.  Maybe I can subclass 
> django.db.backends.postgresql.base.DatabaseWrapper.init_db_connection_state()?
> 
> Also, where can I hook that into my app?  If I subclass that, I’m not sure if 
> I can/should put that in the Model, View, or URL of modules of my app, so 
> that it’s called?  Do I create an entirely new version of 
> django.db.backends.postgresql?
> 
> Ideally this would work with both the basic built-in development server 
> connecting to a local host Postgres, as well as in a uWSGI environment with 
> possible pgBouncer in the stack somewhere.
> 
> Thanks!
> 
> -bobby
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to django-users@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-users 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/06a0a777-9823-4112-9683-131e277d0c2b%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/B3B59F19-C447-41B9-B839-D645FB9F07E4%40gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Scaling Django

2016-02-03 Thread Erik Cederstrand

> Den 3. feb. 2016 kl. 22.30 skrev Joshua Pokotilow :
> 
> At the startup where I work, we've written a lot of our server code in 
> Django. So far, we've adopted a "build it fast" mentality, so we invested 
> very little time in optimizing our code. A small amount of load testing has 
> revealed our codebase / infrastructure as it stands today needs to run faster 
> and support more users.
> 
> We recently hired some new engineers who are extremely skeptical that we 
> should optimize our existing code.

I was in a startup like that. We *had* a working solution, and we *had* 
customers. Not enough to pay our salaries, but enough to keep us and our 
investors hopeful.

Someone decided we needed a rewrite, because Django, because blog posts, 
because WebScale(TM), because in one year we might have 1000-fold users if our 
wildest startup dreams came true. So we started to rewrite. It was supposed to 
take one month. Our working, legacy solution started to deteriorate because we 
were busy rewriting. Two months. Bugs reports piled up in the tracker, but we 
didn't care because the rewrite was just around the corner and would solve 
everything, and we couldn't possibly work on two systems at the same time. Some 
customers left, but it was okay because our WebScale solution would make us 
filthy rich. Three months. Everyone was overworked, tired and the WebScale 
solution was still just around the corner... Four months, and our investors 
decided we were not part of the 1%.

In short, don't rewrite. Refactor. And know *exactly* why you are refactoring. 
As in, "We have profiled, discussed architecture, hardware, algorithms, etc, 
etc. The GIL is killing us and Guido doesn't care", not "Django doesn't scale". 
The year "monolithic" is an argument in itself is the year of the HURD desktop.

Except if you're programming in VBScript. Then by all means, rewrite.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/DC23E544-1108-40A5-821D-681E498DBEAC%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.


Re:

2016-02-03 Thread Denis Bellavance
Hi,

The "remote: Found another file with the destination path" is not actually 
an error, I actually have thousands of those and it is normal. It's simply 
suppose to be a warning that was added in version 1.9:
https://github.com/django/django/commit/5304494585c58b0c9245ea9896a6d6122a8673a2
"Added warning to collectstatic when static files have clashing names"

It should not make the code crash... but somehow maybe the logger become 
unavailable and it is why I get exception "Resource temporarily 
unavailable" ?

I will try turning down verbosity (so shouldn't be logging this anymore), 
and see if it help... (verbosity=0)



On Wednesday, February 3, 2016 at 6:34:47 AM UTC-8, Sergiy Khohlov wrote:
>
>  Hello Denis, 
>
>  This issue  is not related  to the  django-require.  Look like you have 
> two  files (with  same name ) at the different paths. New  collect static 
> is trying to colelct both files and can not decided  which one should be 
> used.  
> Could you please  send part of the section  STATICFILES_DIRS and 
> INSTALLED_APP ? 
>
> Many thanks,
>
> Serge
>
>
> +380 636150445
> skype: skhohlov
>
> On Wed, Feb 3, 2016 at 9:19 AM, Denis > 
> wrote:
>
>> Hi,
>>
>> I'm looking at upgrading my application from 1.6 to 1.9.2... An issue I'm 
>> encountering now is that on Heroku, running collectstatic doesn't work on 
>> 1.9.2. The odd  things is that if I try 1.8.7 instead, it work great...
>>
>>
>> Running:
>> remote: python manage.py collectstatic -i docs -i tests --noinput -v 3
>> remote: Copying '/app/accounts/static/...js'
>> remote: Copying '/app/accounts/static/js'
>> remote: Copying '/app/accounts/static/js'
>> ... [ Thousands of lines... ]
>> remote: Found another file with the destination path 
>> 'app/controllers/...js'. It will be ignored since only the first 
>> encountered file is collected. If this is not what you want, make sure 
>> every static file has a unique path.
>> remote: Found another file with the destination path 
>> 'app/controllers/js'. It will be ignored since only the first 
>> encountered file is collected. If this is not what you want, make sure 
>> every static file has a unique path.
>> remote: Traceback (most recent call last):
>> remote: output = self.handle(*args, **options)
>> remote:   File 
>> "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py",
>>  
>> line 176, in handle
>> remote: collected = self.collect()
>> remote:   File 
>> "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py",
>>  
>> line 114, in collect
>> remote: level=1,
>> remote:   File 
>> "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py",
>>  
>> line 201, in log
>> remote: self.stdout.write(msg)
>> remote:   File 
>> "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py",
>>  
>> line 111, in write
>> remote: self._out.write(force_str(style_func(msg)))
>> remote: IOError: [Errno 11] Resource temporarily unavailable
>>
>>
>> I do have a lot of static file... Unsure that matter.
>>
>> An other things is that it doesn't seem to fail at the same point in the 
>> collectstatic command every time I run it. So look like it's failing on 
>> different file every time I run it...
>>
>>
>> I did find this: https://github.com/etianen/django-require/issues/20
>> But I actually don't use django-require... (It's not in my 
>> requirement.txt of what I install on Heroku)
>>
>>
>> I also see it fail (sometime but much less often as the "Resource 
>> temporarily unavaible" error this way:
>> remote: Found another file with the destination path 'accounts/..html'. 
>> It will be ignored since only the first encountered file is collected. If 
>> this is not what you want, make sure every static file has a unique path.
>> remote: Found another file with the destination path 'accounts/...js'. It 
>> will be ignored since only the first encountered file is collected. If this 
>> is not what you want, make sure every static file has a unique path.
>> remote: Traceback (most recent call last):
>> remote:   File "manage.py", line 10, in 
>> remote: execute_from_command_line(sys.argv)
>> remote: utility.execute()
>> remote: self.execute(*args, **cmd_options)
>>
>> Which is even less helpful in figuring out what's going on...
>>
>>
>> So I'm looking for some though on what this error mean and how to debug 
>> it further. We never encountered that error when on 1.6. Testing on 1.8.7 
>> works fine but then if I only change the Django version (nothing else) to 
>> 1.9.2, I start getting that error...
>>
>>
>> Thanks everyone!
>>
>> Denis Bellavance
>> Peach
>> Seattle
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> ema

Re:

2016-02-03 Thread Denis Bellavance
Allright,

Deploying with verbosity=0 solved my problem.

Something is happening in that  self.log(  call at line 112 that make it 
crash on Heroku. Unsure what it is and unsure the issue is on Django side.

At least this resolves my problem and there is some reference if someone 
else encounter this issue on Heroku too...

Thanks

On Wednesday, February 3, 2016 at 10:36:01 PM UTC-8, Denis Bellavance wrote:
>
> Hi,
>
> The "remote: Found another file with the destination path" is not 
> actually an error, I actually have thousands of those and it is normal. 
> It's simply suppose to be a warning that was added in version 1.9:
>
> https://github.com/django/django/commit/5304494585c58b0c9245ea9896a6d6122a8673a2
> "Added warning to collectstatic when static files have clashing names"
>
> It should not make the code crash... but somehow maybe the logger become 
> unavailable and it is why I get exception "Resource temporarily 
> unavailable" ?
>
> I will try turning down verbosity (so shouldn't be logging this anymore), 
> and see if it help... (verbosity=0)
>
>
>
> On Wednesday, February 3, 2016 at 6:34:47 AM UTC-8, Sergiy Khohlov wrote:
>>
>>  Hello Denis, 
>>
>>  This issue  is not related  to the  django-require.  Look like you have 
>> two  files (with  same name ) at the different paths. New  collect static 
>> is trying to colelct both files and can not decided  which one should be 
>> used.  
>> Could you please  send part of the section  STATICFILES_DIRS and 
>> INSTALLED_APP ? 
>>
>> Many thanks,
>>
>> Serge
>>
>>
>> +380 636150445
>> skype: skhohlov
>>
>> On Wed, Feb 3, 2016 at 9:19 AM, Denis  wrote:
>>
>>> Hi,
>>>
>>> I'm looking at upgrading my application from 1.6 to 1.9.2... An issue 
>>> I'm encountering now is that on Heroku, running collectstatic doesn't work 
>>> on 1.9.2. The odd  things is that if I try 1.8.7 instead, it work great...
>>>
>>>
>>> Running:
>>> remote: python manage.py collectstatic -i docs -i tests --noinput -v 3
>>> remote: Copying '/app/accounts/static/...js'
>>> remote: Copying '/app/accounts/static/js'
>>> remote: Copying '/app/accounts/static/js'
>>> ... [ Thousands of lines... ]
>>> remote: Found another file with the destination path 
>>> 'app/controllers/...js'. It will be ignored since only the first 
>>> encountered file is collected. If this is not what you want, make sure 
>>> every static file has a unique path.
>>> remote: Found another file with the destination path 
>>> 'app/controllers/js'. It will be ignored since only the first 
>>> encountered file is collected. If this is not what you want, make sure 
>>> every static file has a unique path.
>>> remote: Traceback (most recent call last):
>>> remote: output = self.handle(*args, **options)
>>> remote:   File 
>>> "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py",
>>>  
>>> line 176, in handle
>>> remote: collected = self.collect()
>>> remote:   File 
>>> "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py",
>>>  
>>> line 114, in collect
>>> remote: level=1,
>>> remote:   File 
>>> "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py",
>>>  
>>> line 201, in log
>>> remote: self.stdout.write(msg)
>>> remote:   File 
>>> "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/base.py",
>>>  
>>> line 111, in write
>>> remote: self._out.write(force_str(style_func(msg)))
>>> remote: IOError: [Errno 11] Resource temporarily unavailable
>>>
>>>
>>> I do have a lot of static file... Unsure that matter.
>>>
>>> An other things is that it doesn't seem to fail at the same point in the 
>>> collectstatic command every time I run it. So look like it's failing on 
>>> different file every time I run it...
>>>
>>>
>>> I did find this: https://github.com/etianen/django-require/issues/20
>>> But I actually don't use django-require... (It's not in my 
>>> requirement.txt of what I install on Heroku)
>>>
>>>
>>> I also see it fail (sometime but much less often as the "Resource 
>>> temporarily unavaible" error this way:
>>> remote: Found another file with the destination path 'accounts/..html'. 
>>> It will be ignored since only the first encountered file is collected. If 
>>> this is not what you want, make sure every static file has a unique path.
>>> remote: Found another file with the destination path 'accounts/...js'. 
>>> It will be ignored since only the first encountered file is collected. If 
>>> this is not what you want, make sure every static file has a unique path.
>>> remote: Traceback (most recent call last):
>>> remote:   File "manage.py", line 10, in 
>>> remote: execute_from_command_line(sys.argv)
>>> remote: utility.execute()
>>> remote: self.execute(*args, **cmd_options)
>>>
>>> Which is even less helpful in figuring out what's going on...
>>>
>>>
>>