Re: reading CSV file with non ASCII characters

[elcaiaimar]

Thank you for your quick answers. I've tried what you recommended me. My 
code is like that now:

import unicodecsv as csv

csvfile=open("mediosdigitales.csv")
reader = csv.reader(csvfile, encoding='utf-8', delimiter=',')
for row in reader:
titulo=row['titulo']
pais=row['pais']
print titulo
print pais

However, when I compile it I get this error: *list indices must be 
integers, not str*

How could I solve this problem?

Thank you again

El sábado, 6 de febrero de 2016, 3:14:01 (UTC+1), elcaiaimar escribió:
>
> Hello,
>
> I have a CSV File and I want read it. The problem is that it has non ASCII 
> characters such as 'Ñ' and accents and I need that they are recognised to 
> save the CSV content in a DB.
>
> To simplify, I've summed up my code in django to the next:
>
> ​import csv
>
> reader = csv.DictReader(open("file.csv", "rb"))
> for row in reader:
> title=row['title']
> country=row['country']
> print title
> print country
>
> This code returns, for example: Espa�a or Mediterr�neo and I want to get 
> España or Mediterráneo.
>
> Thank you very much!
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/fa59d6cd-588e-4754-aa97-a8ca9a5f6e5e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: reading CSV file with non ASCII characters

[Tim Chase]

On 2016-02-06 03:06, elcaiaimar wrote:
> import unicodecsv as csv
> 
> csvfile=open("mediosdigitales.csv")

> > reader = csv.DictReader(open("file.csv", "rb"))

> reader = csv.reader(csvfile, encoding='utf-8', delimiter=',')
> for row in reader:
> titulo=row['titulo']
> pais=row['pais']
> 
> However, when I compile it I get this error: *list indices must be 
> integers, not str*

you switched from using csv.DictReader() to csv.reader()

Just switch back and your string-indexing should be good.

-tkc



-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20160206063817.2ec806cf%40bigbox.christie.dr.
For more options, visit https://groups.google.com/d/optout.

django現在インストールエラーにつきまして

[makamaka32]

お世話になります。
現在djangoを学習しようとしている者ですがインストール時にエラーが起こり先に進めないようになっております。

easy_installにてdjangoをダウンロードし
python manage.py startapp myapp まで進むと
importError: No module named mysite.settings
と表示されます。

こういった場合どのように対処すればよろしいでしょうか？
同じフォルダの中にはsettings.pyは存在するのですが。

誠に申し訳ございません。
回答どうかよろしくお願いいたします。

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/9a31b441-6771-4078-844f-90da754bfd52%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Add model item with AJAX pop up

[guettli]

 

I use django-select2  for easy 
ForeignKey picking. Works nice.


But how can I add a new item in the ForeignKey table?


I would like to have a big [+] button near the select2 field and then a 
(ajax) modal dialog should open (like in the admin interface).


AFAIK django-select2 does not help here. Is there a different library which 
can help here?


I would like to re-use and not to re-invent. Maybe I was blind, but I could 
not

find a matching comparison grid on djangopackages.com.


I would like to avoid a "old-school" browser-window-pop-up :-)


Regards,

  Thomas Güttler

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/f7c5e464-61f0-4c0b-a980-7bfa553d1b67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Django and process uploading file

[Tino Fischer]

Simplest way to change the cursor with javascript:

document.body.style.cursor = 'wait';
or
document.body.style.cursor = 'progress';


When done loading:

document.body.style.cursor = 'auto';

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/9679158a-017e-4360-8c91-cb3c6573f430%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django and process uploading file

[ylativ oknesyl]

Thanks

С уважением В. Лысенко




2016-02-06 16:02 GMT+02:00 Tino Fischer :

> Simplest way to change the cursor with javascript:
>
> document.body.style.cursor = 'wait';
> or
> document.body.style.cursor = 'progress';
>
>
> When done loading:
>
> document.body.style.cursor = 'auto';
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-users/unB4MKiVIik/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/9679158a-017e-4360-8c91-cb3c6573f430%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAKpbR8kGXBarYeDjT-Gm84%3DPOh-WDDb9e498%2BCCFqs7YKe%3DiVw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Best free audio/video (crossbrowser) player for website

[ylativ oknesyl]

Projekktor not suitable (tasted)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/a0961076-e3b0-473d-818c-dbcc11259958%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: reading CSV file with non ASCII characters

[paul . hermeneutic]

On Sat, Feb 6, 2016 at 5:38 AM, Tim Chase
 wrote:
> On 2016-02-06 03:06, elcaiaimar wrote:
>> import unicodecsv as csv
>>
>> csvfile=open("mediosdigitales.csv")
>
>> > reader = csv.DictReader(open("file.csv", "rb"))
>
>> reader = csv.reader(csvfile, encoding='utf-8', delimiter=',')

Do you actually know that the character encoding of the file is utf_8?

If the file is coming from a Western European (including English)
system, it could easily be cp_1252.

https://docs.python.org/3/library/codecs.html

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAJ4%2B4apZPFzpYg6gqynchVt9j5URVfAT0pPrvX9V8s%3Dj-JmPyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

django social auth

[ahmed waqas Nasir]

i am new in django i am trying to add thirdparty verification using social 
auth

after configuration when i run migrations i got following error

InDjango110Warning: SubfieldBase has been deprecated. Use 
Field.from_db_value i stead. RemovedInDjango110Warning)

i know the reason of this error is because i am using django 1.9 and there 
is some fucntion or something which is depricated in 1.9 and that is 
related to database..
but  i dont know how to solve this problem

can any one help?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/05758f04-94ba-46fc-bc0e-39ce6ec00182%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Scaling Django

[Dexter T.]

Lots of great replies already.
I also want to add a few (random) things ...

- have you clearly defined and isolated what issue(s) are you facing?
- you mentioned using DRF in a service, with a large JSON reponse taking 
seconds to finish, how did you troubleshoot/profile this? Seconds to process 
server-side? Seconds to download client-side? Where specifically? If you said 
you don't know, then find out!
- your system will have so many legs, have you made an effort to instrument and 
measure and isolate which parts are slow and why?
- you mentioned using the debug toolbar, have you proven that your database 
schema is optimal? Any queries in your slow queries log? Indexes used and ok 
and optimal? For your workload, can_read caching help? Db replicas be of help?
- how are your server resources utilized? Are you sure you are not bottlenecked 
by thrashing disk-io? Overcomitted CPU? Low memory/swapping? File descriptor 
count?
- have you checked if clients are not bottlenecked? An ajax call to download a  
complex nested json object is both costly to serialize, CPU and bandwidth wise. 
Gzip can help here, if applicable.
- for more context, can you share some numbers, like http and db level req/sec, 
row count for the most heavily used tables? How about server infrastructure 
specs?

Note that these are basic questions and are basic problem-solving steps, im 
assuming your teams should be aware and be taking steps like these already.

In one project of mine, we're doing a 100gb mysql db, some tables above 100mil 
recs and growing rapidly, properly indexed and optimized, it works ok on a 
lowly single vps instance with 8gb ram; workload is clearly oltp, we're 
throwing more sustained writes (100s/sec) than reads, all queries were 
scrutinized, almost all using the ORM, some handwritten SQL, other complex 
queries rewritten to be done at application level, joins are harder at this 
scale and therefore preferrably avoided (major architectural decision 
anticipated). But still we can easily throw hardware if needed. 

For us, scaling is an continuous commitment to measure and refactor.

And one very important learning for me in my years of writing software: 
rewriting is very very very very costly. 

These new engineers/other colleagues coming in, are they familiar with the 
domain problem, the exisiting codebase, the scale at which you operate now and 
expected in the future? Are they experienced in doing similar scaling before? 
And even if you think you can throw your old work, and now that you guys think 
you know better, be very careful of The-Second-System-Effect.

I hope you succeed.


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/ddc6db79-af4c-4e78-a16f-84f2dc8b69ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add model item with AJAX pop up

[guettli]

The result should look like the "create user" modal dialog on this page:

https://jqueryui.com/dialog/#modal-form

Any idea how to get this done?

Up to now I use the usual pattern:

 - redirect after post, if the submitted data is valid.
 - show form again with error message if the data is not valid.

I could code this myself. But I ask myself it is good to do so.

I guess this has already been solved many times before. I most cases
there is a library/reusable-app. But here I found none.

Regards,
  Thomas Güttler

Am Samstag, 6. Februar 2016 14:12:48 UTC+1 schrieb guettli:
>
> I use django-select2  for 
> easy ForeignKey picking. Works nice.
>
>
> But how can I add a new item in the ForeignKey table?
>
>
> I would like to have a big [+] button near the select2 field and then a 
> (ajax) modal dialog should open (like in the admin interface).
>
>
> AFAIK django-select2 does not help here. Is there a different library 
> which can help here?
>
>
> I would like to re-use and not to re-invent. Maybe I was blind, but I 
> could not
>
> find a matching comparison grid on djangopackages.com.
>
>
> I would like to avoid a "old-school" browser-window-pop-up :-)
>
>
> Regards,
>
>   Thomas Güttler
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/db6f6d27-727b-4dd0-aae6-848ee94e528f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Scaling Django

[Sergiy Khohlov]

Print database structure.
Check possibility of DB normalization.
100 GB  (my "record" is 452 GB  )is not so high but  this size requires
some attention. (Look like you Mysql used only one db file: try to set
table per file.  Check index size , and verify that indexes  are working
corectly)
Review  your project:
 try to avoid  Many to Many field
Is it possible switch from hardcode SQL to  stored function and procedure ?
 Look like  this issue in not connected to django only.


Many thanks,

Serge


+380 636150445
skype: skhohlov

On Sat, Feb 6, 2016 at 7:09 PM, Dexter T.  wrote:

> Lots of great replies already.
> I also want to add a few (random) things ...
>
> - have you clearly defined and isolated what issue(s) are you facing?
> - you mentioned using DRF in a service, with a large JSON reponse taking
> seconds to finish, how did you troubleshoot/profile this? Seconds to
> process server-side? Seconds to download client-side? Where specifically?
> If you said you don't know, then find out!
> - your system will have so many legs, have you made an effort to
> instrument and measure and isolate which parts are slow and why?
> - you mentioned using the debug toolbar, have you proven that your
> database schema is optimal? Any queries in your slow queries log? Indexes
> used and ok and optimal? For your workload, can_read caching help? Db
> replicas be of help?
> - how are your server resources utilized? Are you sure you are not
> bottlenecked by thrashing disk-io? Overcomitted CPU? Low memory/swapping?
> File descriptor count?
> - have you checked if clients are not bottlenecked? An ajax call to
> download a  complex nested json object is both costly to serialize, CPU and
> bandwidth wise. Gzip can help here, if applicable.
> - for more context, can you share some numbers, like http and db level
> req/sec, row count for the most heavily used tables? How about server
> infrastructure specs?
>
> Note that these are basic questions and are basic problem-solving steps,
> im assuming your teams should be aware and be taking steps like these
> already.
>
> In one project of mine, we're doing a 100gb mysql db, some tables above
> 100mil recs and growing rapidly, properly indexed and optimized, it works
> ok on a lowly single vps instance with 8gb ram; workload is clearly oltp,
> we're throwing more sustained writes (100s/sec) than reads, all queries
> were scrutinized, almost all using the ORM, some handwritten SQL, other
> complex queries rewritten to be done at application level, joins are harder
> at this scale and therefore preferrably avoided (major architectural
> decision anticipated). But still we can easily throw hardware if needed.
>
> For us, scaling is an continuous commitment to measure and refactor.
>
> And one very important learning for me in my years of writing software:
> rewriting is very very very very costly.
>
> These new engineers/other colleagues coming in, are they familiar with the
> domain problem, the exisiting codebase, the scale at which you operate now
> and expected in the future? Are they experienced in doing similar scaling
> before? And even if you think you can throw your old work, and now that you
> guys think you know better, be very careful of The-Second-System-Effect.
>
> I hope you succeed.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/ddc6db79-af4c-4e78-a16f-84f2dc8b69ae%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CADTRxJN1yiVH1oviDiJRKCL2dN5%2BKpTwxyU7SksrF%3DwV6sWPXA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: django現在インストールエラーにつきまして

[Sergiy Khohlov]

 look like you have used pip or easy_install  for downloading  django
 Verify that  django is ok

python -c "import django; print(django.__path__)"

After this

django-admin startproject mysite

 this command creates mysite dir  and  settings.py of course


Many thanks,

Serge


+380 636150445
skype: skhohlov

2016-02-06 9:37 GMT+02:00 makamaka32 :

> お世話になります。
> 現在djangoを学習しようとしている者ですがインストール時にエラーが起こり先に進めないようになっております。
>
> easy_installにてdjangoをダウンロードし
> python manage.py startapp myapp まで進むと
> importError: No module named mysite.settings
> と表示されます。
>
> こういった場合どのように対処すればよろしいでしょうか？
> 同じフォルダの中にはsettings.pyは存在するのですが。
>
> 誠に申し訳ございません。
> 回答どうかよろしくお願いいたします。
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/9a31b441-6771-4078-844f-90da754bfd52%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CADTRxJPC4sZrb_xD%2BumARGpNseukNKc%2Brc7Tw_Hx24iCjpbP8g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: django現在インストールエラーにつきまして

[Yoong Kang Lim]

(日本語で話すのが久しぶりだけど... 笑)

"mysite" 
の中に"settings.py"と"__init__.py"がありますか？その二つのファイルがなければ、importするとerrorが起こります。
 

On Saturday, February 6, 2016 at 11:56:13 PM UTC+11, makamaka32 wrote:
>
> お世話になります。
> 現在djangoを学習しようとしている者ですがインストール時にエラーが起こり先に進めないようになっております。
>
> easy_installにてdjangoをダウンロードし
> python manage.py startapp myapp まで進むと
> importError: No module named mysite.settings
> と表示されます。
>
> こういった場合どのように対処すればよろしいでしょうか？
> 同じフォルダの中にはsettings.pyは存在するのですが。
>
> 誠に申し訳ございません。
> 回答どうかよろしくお願いいたします。
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/a1663d2c-900b-48af-9d15-46b1fd0eee15%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django formtools wizard produces blank page (No error returned)

[Martín Torre Castro]

Solved. Silly error. The name for the template function in the wizard
should be *get_template_names* and it was *get_templates_name*.

Thank you anyway!!!

On 2 February 2016 at 10:17, Martín Torre Castro <
martin.torre.cas...@gmail.com> wrote:

> Yes, those two templates are both entirely the code posted.
>
> On 2 February 2016 at 10:07, James Schneider 
> wrote:
>
>>
>>
>> On Tue, Feb 2, 2016 at 12:26 AM, Martín Torre Castro <
>> martin.torre.cas...@gmail.com> wrote:
>>
>>> I have DEBUG = True in my settings. When making the request the 200 http
>>> code appears. The 500 "Internal server error" code only appears when trying
>>> to obtain info through the django-debug-toolbar
>>>
>>>
>> In that case you should be seeing something in your web server logs
>> regarding the internal server error. I'd be quite surprised if the debug
>> toolbar was the cause of the error. It must be something being set
>> somewhere that it isn't expecting.
>>
>>
>> I went back through your code posting, and I was curious about something:
>>
>> *registration/test_step1.html*
>>
>> Two fields form
>> 
>> 
>>
>>
>> *registration/test_step2.html*
>> Message form
>> 
>>
>> Are these your actual templates? I originally made an assumption that you
>> had intentionally omitted the {% block %} and {% extends %} tags, but now
>> I'm wondering if these are the actual templates and what you are receiving
>> when you see a blank page is actually just the raw HTML that you posted
>> above (check the source of the blank page)?
>>
>> Just a thought.
>>
>> -James
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Django users" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/django-users/mAi_fB_MTwo/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> django-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-users@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/CA%2Be%2BciXf7XHLEQeT_Onqx_Zq%3DKBmOTofXayWN4SoaxtHMhshvA%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAKijOkyxhsdjR8MR6%3DWruq%2BiL8yutS0mB_J51hQR0yH%2Bp0JMOg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Scaling Django

[Dexter T.]

Hi Sergiy, are you referring to my post or to the OP?

On Sunday, February 7, 2016 at 6:03:11 AM UTC+8, Sergiy Khohlov wrote:
>
> Print database structure.
> Check possibility of DB normalization. 
>

You might have meant "denormalization" here (?), especially when operating 
at such scale. We do used denormalization for some of our larger tables.
 

> 100 GB  (my "record" is 452 GB  )is not so high but  this size requires 
> some attention. (Look like you Mysql used only one db file: try to set 
> table per file.  Check index size , and verify that indexes  are working 
> corectly)
>

We are using innodb_file_per_table. But see that I mentioned that all this 
100GB data fit on a lowly 8GB ram VM, 50% of which was allocated to innodb 
buffers. With such little resources, but at the same time intimately 
knowing your database workload, it is still possible to handle such db 
size. And yes, our indexes are used well, as most queries were EXPLAINed 
and optimized accordingly. 

What hardware are you running your 452GB db in?

Review  your project: 
>  try to avoid  Many to Many field 
> Is it possible switch from hardcode SQL to  stored function and procedure 
> ? 
>

See my above post about denormalization. And arguably storedprocs are even 
harder to manage, code-wise, and deployment wise.
 

>  Look like  this issue in not connected to django only. 
>

Again, if you are referring to my post, I am not the OP. Not that our 
system is perfect, and yes we're not the ones with scaling problems. 
I was in fact sharing the practices of scaling that worked for us. See the 
OPs post on what problems they're facing (organizational / political / 
methodological).

Cheers!
 

>
>
> Many thanks,
>
> Serge
>
>
> +380 636150445
> skype: skhohlov
>
> On Sat, Feb 6, 2016 at 7:09 PM, Dexter T. 
> > wrote:
>
>> Lots of great replies already.
>> I also want to add a few (random) things ...
>>
>> - have you clearly defined and isolated what issue(s) are you facing?
>> - you mentioned using DRF in a service, with a large JSON reponse taking 
>> seconds to finish, how did you troubleshoot/profile this? Seconds to 
>> process server-side? Seconds to download client-side? Where specifically? 
>> If you said you don't know, then find out!
>> - your system will have so many legs, have you made an effort to 
>> instrument and measure and isolate which parts are slow and why?
>> - you mentioned using the debug toolbar, have you proven that your 
>> database schema is optimal? Any queries in your slow queries log? Indexes 
>> used and ok and optimal? For your workload, can_read caching help? Db 
>> replicas be of help?
>> - how are your server resources utilized? Are you sure you are not 
>> bottlenecked by thrashing disk-io? Overcomitted CPU? Low memory/swapping? 
>> File descriptor count?
>> - have you checked if clients are not bottlenecked? An ajax call to 
>> download a  complex nested json object is both costly to serialize, CPU and 
>> bandwidth wise. Gzip can help here, if applicable.
>> - for more context, can you share some numbers, like http and db level 
>> req/sec, row count for the most heavily used tables? How about server 
>> infrastructure specs?
>>
>> Note that these are basic questions and are basic problem-solving steps, 
>> im assuming your teams should be aware and be taking steps like these 
>> already.
>>
>> In one project of mine, we're doing a 100gb mysql db, some tables above 
>> 100mil recs and growing rapidly, properly indexed and optimized, it works 
>> ok on a lowly single vps instance with 8gb ram; workload is clearly oltp, 
>> we're throwing more sustained writes (100s/sec) than reads, all queries 
>> were scrutinized, almost all using the ORM, some handwritten SQL, other 
>> complex queries rewritten to be done at application level, joins are harder 
>> at this scale and therefore preferrably avoided (major architectural 
>> decision anticipated). But still we can easily throw hardware if needed.
>>
>> For us, scaling is an continuous commitment to measure and refactor.
>>
>> And one very important learning for me in my years of writing software: 
>> rewriting is very very very very costly.
>>
>> These new engineers/other colleagues coming in, are they familiar with 
>> the domain problem, the exisiting codebase, the scale at which you operate 
>> now and expected in the future? Are they experienced in doing similar 
>> scaling before? And even if you think you can throw your old work, and now 
>> that you guys think you know better, be very careful of 
>> The-Second-System-Effect.
>>
>> I hope you succeed.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com .
>> To post to this group, send email to django...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this disc