middleware response processing always occurs

[George Lund]

We recently uncovered a bug in our code, which was pretty confusing,
and is explainable only via the fine-print of
http://docs.djangoproject.com/en/dev/topics/http/middleware/ -- and
reading through the Django source.  So I thought I'd share, and maybe
a Django developer might comment on whether the current behaviour is
really desirable, and if so perhaps consider improving the docs.

We have some Middleware that updates the user's session, from a
process_response method.  Sometimes this would fail, because the
request object had no session.

It turns out this would happen because the Common middleware had
issued a redirect (specifically, APPEND_SLASH was redirecting us to
correct the view's URL).

In a case like this, the current behaviour is not like the 'onion' as
illustrated in the docs, because whilst the redirect meant that *only*
the Common middleware had its process_request function executed, every
loaded middleware's process_response is executed.  (This is mentioned
in the docs, it turns out, in the process_request section but not in
the process_response section.)

What this means is that even though it seems, and is the case 99% of
the time, that middleware listed later can rely totally on middleware
further up the list (i.e. the 'onion' model), this is in fact not the
case, for processing responses.  So response processing code is either
subject to rare bugs, or has to contain extra checks to make sure of
its preconditions (in our case a check for hasattr(request,
'session') ).

Personally I think this is confusing enough that it would be better if
the program logic could be changed, so that later-listed middleware
would definitely only either run, or not run, depending on whether the
earlier middleware allows the request to proceed or not.  But I guess
such a change could break existing code, so at the very least the docs
could use a bit more explanation and or highlighting in bold!

thanks

George Lund


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

traceback - no local variables

[George Lund]

We have a live server crash issues for which we are receiving emails
from Django from our server, which is great.

But I'm having trouble debugging the problem because the local
variables don't seem to be included in the email.  All I can see is
call stack followed by a  object - but nothing else.

Does anyone know how to make Django supply all the variables in the
email?

TIA
George Lund

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: traceback - no local variables

[George Lund]

That certainly looks more useful than the default Django behaviour -
thanks very much.

(I haven't tried it yet - but I assume I need to disable the standard
Django email if I use this, by removing the list of ADMINs. I wonder
if that will have any other consequences?)




On Mar 12, 7:13 am, Thomas Guettler  wrote:
> Hi,
>
> I attached a HandleExceptionMiddleware I use. You need to modify it
> to your needs. It attaches the html page of cgitb and the django
> debug page to the email.
>
>   Thomas
>
> George Lund schrieb:
> ...
> > Does anyone know how to make Django supply all the variables in the
> > email?


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

mysql warnings raise exception

[George Lund]

I notice that this problem gets discussed occasionally but with no proper 
resolution, so I felt obliged to raise it again, as the consequences are 
quite serious for MySQL users.

When MySQL issues a warning, Django sometimes raises an exception. This is 
discussed in https://code.djangoproject.com/ticket/12293, which is marked 
as wontfix.

The main problem is that the behaviour is different depending whether DEBUG 
is True or not. This leads to problems with testing and deployment that in 
an extreme case could cause data loss.

A secondary problem is that the exception in question isn't being caught 
and re-thrown as a Django exception, so that code that wants to deal with 
such scenarios must catch a MySQL-specific exception, tying us to the one 
database back-end.

For me, the right choice would be to make this configurable, and completely 
independent of the DEBUG setting. The right default would be to ignore 
warnings, as in general that's how most programs coding against MySQL would 
expect to treat them. Most people I would argue would expect a warning to 
be ignored and for processing to continue.

*Please* can this wontfix be re-considered -- the interwebs appear to be 
full of people getting caught out by this (if it will help the case, I can 
list numerous references!)

TIA

George

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/Zj13E62NFOYJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Bulk delete - performance / object collection

[George Lund]

I'm trying to bulk-delete several million rows from my database.

The docs for Django 
1.3say
 "this will, whenever possible, be executed purely in SQL". A pure-SQL 
delete is what I want in this case, so that's fine.

However, the code is never getting as far as running any SQL.

Interrupting the script shows that the delete method on the QuerySet is 
trying to use a "Collector" to construct model instances for each row I'm 
trying to delete. This is going to take too long (and may in fact consume 
all the memory available) -- I don't think it's practical to wait in this 
case. (I've tried waiting over half an hour!)

(I'm looking at django.db.models.query.QuerySet.delete and 
django.db.models.deletion.Collector.collect 
/ Collector.add.)

What's the point in doing the delete "purely in SQL" if all of the objects 
are getting constructed anyway? Why do they need to be "collected" before 
the SQL DELETE is run? The model instance in this case has no child rows, 
to which the delete might need to be cascaded.

Meanwhile I can construct the SQL by hand easily enough, but I feel this 
isn't doing things the right way.

thanks for any help

George

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/W4LqKzcnlaYJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Bulk delete - performance / object collection

[George Lund]

>
> > Meanwhile I can construct the SQL by hand easily enough, but I feel this 
> > isn't doing things the right way. 
>
> I've had this discussion with other developers. Many feel they need to 
> slavishly adhere to the ORM and if you don't you're some type of evil 
> entity. I could not disagree more. Django and the ORM are tools, and 
> should be used only when they are the right tools for the job. I'd 
> just go ahead and do the delete in SQL.
>

I don't think SQL is evil, but there are lots of reasons why mixing 
programming languages (and abstraction levels) to solve a problem isn't 
really ideal.

>From a very personal point of view I'd like to see archaic languages like 
SQL go away for ever one day, in the meantime I completely agree that if 
the ORM doesn't do what's needed then the right choice is to go with 
whatever tools will :-)

thanks
George

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/x6y5ORLtg3wJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Bulk delete - performance / object collection

[George Lund]

Thank you very much for this. I'll catch up with those threads and read 
more about DSE, which looks really interesting work.

regards

George

On Wednesday, 2 January 2013 11:50:54 UTC, Cal Leeming [Simplicity Media 
Ltd] wrote:
>
> Hi George,
>
> This is one area I spent quite a lot of time in personally, see;
>
> https://groups.google.com/forum/?fromgroups=#!msg/django-users/iRhWD0FtW8k/0KAMF3ub-ZYJ
> https://groups.google.com/forum/#!topic/django-users/hgLrwMoFLII
>
> Bulk operations using the ORM isn't always the right thing to do - and it 
> entirely depends on what you consider bulk and acceptable performance.
>
> You might want to look at the source code for this, to see how they handle 
> bulk operations (they implemented the same bulk_update approach mentioned 
> in the above threads)
> http://pypi.python.org/pypi/dse
>
> Although bypassing the ORM might feel wrong at first, sometimes it is 
> completely acceptable - you just need to make sure you don't abuse/misuse 
> it unnecessarily.
>
> Cal
>
> On Wed, Jan 2, 2013 at 11:29 AM, George Lund 
> > wrote:
>
>> I'm trying to bulk-delete several million rows from my database.
>>
>> The docs for Django 
>> 1.3<https://docs.djangoproject.com/en/1.3/topics/db/queries/#deleting-objects>say
>>  "this will, whenever possible, be executed purely in SQL". A pure-SQL 
>> delete is what I want in this case, so that's fine.
>>
>> However, the code is never getting as far as running any SQL.
>>
>> Interrupting the script shows that the delete method on the QuerySet is 
>> trying to use a "Collector" to construct model instances for each row I'm 
>> trying to delete. This is going to take too long (and may in fact consume 
>> all the memory available) -- I don't think it's practical to wait in this 
>> case. (I've tried waiting over half an hour!)
>>
>> (I'm looking at django.db.models.query.QuerySet.delete and 
>> django.db.models.deletion.Collector.collect 
>> / Collector.add.)
>>
>> What's the point in doing the delete "purely in SQL" if all of the 
>> objects are getting constructed anyway? Why do they need to be "collected" 
>> before the SQL DELETE is run? The model instance in this case has no 
>> child rows, to which the delete might need to be cascaded.
>>
>> Meanwhile I can construct the SQL by hand easily enough, but I feel this 
>> isn't doing things the right way.
>>
>> thanks for any help
>>
>> George
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msg/django-users/-/W4LqKzcnlaYJ.
>> To post to this group, send email to django...@googlegroups.com
>> .
>> To unsubscribe from this group, send email to 
>> django-users...@googlegroups.com .
>> For more options, visit this group at 
>> http://groups.google.com/group/django-users?hl=en.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/q1ZxexzIdX8J.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Model inheritance problem

[George Lund]

> class MyBaseModel(models.NodeModel):
>name=StringProperty()
>class Meta:
>abstract = True
>
>
In Django you would need CharField or similar. StringProperty is a Google 
App Engine thing? Cf 
http://stackoverflow.com/questions/6132695/django-module-object-has-no-attribute-stringproperty

So your base model has no field called "name".

HTH

George

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Model inheritance problem

[George Lund]

Ah okay, well I've never used that, I thought you were implying this could 
be a general Django question.

Maybe neo4django doesn't support model inheritance -- I took a look at 
their site briefly and couldn't see anything about it.

Sorry not to be more help...

On Tuesday, 17 September 2013 15:28:08 UTC+1, Antonio Mignolli wrote:
>
> Hmm, no, it does not work even without indexed=True, 
> my mistake.
> So the "issue" remains.
>
> Il giorno martedì 17 settembre 2013 16:17:04 UTC+2, Antonio Mignolli ha 
> scritto:
>>
>> Thanks, George, but as I said in the beginning, I'm using neo4django, 
>> which surely has StringProperty, otherwise I would have an 
>> AttributeError on StringProperty, which I don't have.
>>
>> I should have written:
>> from neo4django.db import models
>>
>> and 
>> ...
>> name = models.StringProperty(indexed=True)
>>
>> for completeness.
>>
>> I think I just discovered another neo4django issue,
>> in fact it works perfectly only if such attribute 
>> is not defined with indexed = True.
>>
>>
>> Il giorno martedì 17 settembre 2013 14:27:34 UTC+2, George Lund ha 
>> scritto:
>>>
>>>
>>> class MyBaseModel(models.NodeModel):
>>>>name=StringProperty()
>>>>class Meta:
>>>>abstract = True
>>>>
>>>>
>>> In Django you would need CharField or similar. StringProperty is a 
>>> Google App Engine thing? Cf 
>>> http://stackoverflow.com/questions/6132695/django-module-object-has-no-attribute-stringproperty
>>>
>>> So your base model has no field called "name".
>>>
>>> HTH
>>>
>>> George
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Model with several ManyToManyFields - related_name problem

[George Lund]

I have a model with more than one (3 in fact) ManyToManyFields which
point to the same other model.

As long as I specify distinct related_names for each, all is well.

But the documentation [1] suggests that if I don't need the backwards
relation, then specifying '+' as the related_name should work.  But I
don't seem to be able to specify '+' for more than one of the fields:
I get a 'reverse query clashes' error.

Is this expected?  It feels like a bug (i.e. I'd expect all fields
with related_name="+" to be ignored for this check).

TIA

George


[1: 
http://docs.djangoproject.com/en/dev/ref/models/fields/#django.db.models.ForeignKey.related_name
]

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Model with several ManyToManyFields - related_name problem

[George Lund]

Hiya, thanks for the reply.

So in this case the extra fields are actually denormalized subsets of the 
original field, and the reverse relation can use the main field.  (At least, 
at the moment, that's my plan, but that could change.  Which I guess goes to 
show how rare this case probably actually is.)

But you can see that with the data being so similar on each many-to-many 
table, I'd want to avoid confusing the other model with a set of extra 
properties that aren't so useful and will have similar names!  So I guess 
there might be other cases where you likewise didn't want to create a 
confusing backwards relation, and the '+' behaviour might be expected to 
work as the documentation suggests (to me) that it might.

Incidentally I'm a bit confused about the naming of ManyToMany fields: the 
examples tend to name them in the plural
sites = models.ManyToManyField(Site, verbose_name="list of sites")
but the backwards relation gets created with a singular name (inevitably I 
guess, because that's what the system can work out).  It seems a bit 
inconsistent?

cheers
George

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

django-admin commands from installed egg

[George Lund]

Hi all.. I'm having trouble with Django not registering custom django-admin 
commands from applications that are installed to egg files.

The app is correctly listed in INSTALLED_APPS, and is installed to a 
location on the pythonpath.  But its custom admin commands do not appear 
when running manage.py.

Regular imports (i.e. calling any functions/models in the app) work just 
fine.  The mechanism Django uses to find the custom admin commands is the 
imp module.

So in this case (from an interpreter session started with manage.py shell):

>>> import denorm
>>> denorm


works just fine, but

>>> import imp
>>> imp.find_module("denorm")
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named denorm

does not.  Explicitly mentioning the egg file doesn't help:

>>> imp.find_module("denorm", 
"[snip]/lib/python/site-packages/django_denorm-0.2.0-py2.5.egg")
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No frozen submodule named 
[snip]/lib/python/site-packages/django_denorm-0.2.0-py2.5.egg.denorm

We are running Python 2.5.2 on FreeBSD.

Is there are known issue with the imp module on this version of Python that 
anyone knows about?

Our workaround for now is to install this particular app from source rather 
than as an egg file using setup.py, but this seems weird.

I guess most likely is that there are certain circumstances where imp is 
supposed to work differently from a regular import, but I can't see from the 
docs what those might be.

Any ideas appreciated.

thanks
George Lund

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: unicode question

[George Lund]

I can't explain the additional space, but URL-encoding a Unicode string 
isn't generally possible, it needs to be encoded first.  UTF-8 is almost 
always the correct encoding to use, so in your case:
 qk = quote(query_text.encode('utf8'))

Anything you can do to avoid having to include text like that in a URL (e.g. 
maybe it could be saved to a model instance, and recovered via an id or a 
slug?) would often be a better option.

Hope that helps
George

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.