Would you mind checking if the bug occurs in Django 1.5? If it doesn't, it's a regression introduced by the new transaction management in Django 1.6, and it's a release blocker.
-- Aymeric. On 25 juin 2013, at 22:30, Yo-Yo Ma <[email protected]> wrote: > I should actually note, this bug affects all versions of Postgres, and > presumably all other supported RDBMSs as well. > > On Sunday, June 23, 2013 7:35:45 PM UTC-4, Yo-Yo Ma wrote: > Minor correction: I placed Atomic.__exit__ to verify - the transaction is > commited every time *starting* on the second object (the third stack printed > in the previous post) - it happens at > https://github.com/django/django/blob/master/django/db/transaction.py#L288. > > > On Sunday, June 23, 2013 7:24:40 PM UTC-4, Yo-Yo Ma wrote: > Hi again Russell, > > I did a little digging. I'm not sure, but I may have uncovered the problem. A > transaction block (using `commit_on_success_unless_managed`) is entered and > exited during each fixture object loaded, due to the calls to the > aforementioned method that exist in various model methods (namely, > `save_base`, in this case). Because of this, the transaction is committed > immedately after each object is loaded, despite the attempt to wrap > `commit_on_success_unless_managed` around the context of the `loaddata` call > in the management command. > > The following are the results of my placing print statements (I know that's > old-school - pdb is just too time consuming) inside > `commit_on_success_unless_managed`. In each call, I added: > > print 'AUTOCOMMIT', connection.autocommit > print 'IN ATOMIC BLOCK', connection.in_atomic_block > for frame in inspect.stack(): > print frame[1], frame[3], frame[2] > > as well as a print after the stack of whether atomic() was returned or > _transaction_func() was returned (for easier reading): > > > AUTOCOMMIT False > IN ATOMIC BLOCK False > > django/db/transaction.py commit_on_success_unless_managed 492 > django/core/management/commands/loaddata.py handle 53 > django/core/management/base.py execute 283 > django/core/management/base.py run_from_argv 240 > django/core/management/__init__.py execute 392 > django/core/management/__init__.py execute_from_command_line 399 > manage.py <module> 10 > > ----RETURNING TRANSACTION FUNC > > =========================================================== > > AUTOCOMMIT False > IN ATOMIC BLOCK False > > django/db/transaction.py commit_on_success_unless_managed 492 > django/db/models/base.py save_base 573 > django/core/serializers/base.py save 165 > django/core/management/commands/loaddata.py process_dir 225 > django/core/management/commands/loaddata.py load_label 169 > django/core/management/commands/loaddata.py loaddata 102 > django/core/management/commands/loaddata.py handle 54 > django/core/management/base.py execute 283 > django/core/management/base.py run_from_argv 240 > django/core/management/__init__.py execute 392 > django/core/management/__init__.py execute_from_command_line 399 > manage.py <module> 10 > > ----RETURNING TRANSACTION FUNC > > =========================================================== > > SAVEPOINT False > AUTOCOMMIT True > IN ATOMIC BLOCK False > > ||||||||||||||||||||||||||||||||||||||||||||||| > django/db/transaction.py commit_on_success_unless_managed 492 > django/db/models/base.py save_base 573 > django/core/serializers/base.py save 165 > django/core/management/commands/loaddata.py process_dir 225 > django/core/management/commands/loaddata.py load_label 169 > django/core/management/commands/loaddata.py loaddata 102 > django/core/management/commands/loaddata.py handle 54 > django/core/management/base.py execute 283 > django/core/management/base.py run_from_argv 240 > django/core/management/__init__.py execute 392 > django/core/management/__init__.py execute_from_command_line 399 > manage.py <module> 10 > > ----RETURNING ATOMIC > > =========================================================== > > > The remaining calls were exactly like call 3 (including "IN ATOMIC BLOCK > False", despite the 3rd call having returned `atomic()`). My prima facie > opinion is that `with atomic()` is needed in `loaddata`, instead of `with > commit_on_success_unless_managed`, since the latter acts funky when nested > calls occur (as see in save_base in the stacks printed above). However, the > issue might be something that needs to be resolved in the > transitioning-to-atomic code. I don't fully understand all of this yet, but > it's a start. > > > On Friday, June 21, 2013 4:34:14 PM UTC-4, Yo-Yo Ma wrote: > Pardon one typo: I meant `python manage.py loaddata test_data` in my previous > post. > > On Friday, June 21, 2013 4:32:33 PM UTC-4, Yo-Yo Ma wrote: > Hi Russel, > > Thanks for taking the time to explain that. I tried that same day to > reproduce the issue in a testing env with the simplified models I typed > above, but my hosting provider had some erroneous networking nonsense that > ruined my test after I spent a couple hours setting everything up. I figured > I'm come back to it... and here I am. > > I didn't set up an entire test env and test app this time, just made a fresh > database and ran my apps fixtures on it, but I did test my app again, using a > fresh database without any data. The models and fixtures for which are as > follows (minus most of the decimals, chars, and other non-FK-type fields, > none of which should be related to this problem): > > > # account/models.py > class Account(models.Model): > name = models.CharField(_(u'name'), max_length=255) > > > # orders/models.py > class Order(models.Model): > account = models.ForeignKey('account.Account', verbose_name=_(u'account')) > number = models.IntegerField(_(u'number')) > bill_address = models.OneToOneField( > 'orders.OrderAddress', > null=True, > on_delete=models.SET_NULL, > related_name='bill_address_order', > verbose_name=_(u'bill to address') > ) > > class OrderAddress(models.Model): > account = models.ForeignKey('account.Account', verbose_name=_(u'account')) > order = models.ForeignKey('orders.Order', verbose_name=_(u'order')) > country = models.CharField(_(u'country'), max_length=2) > > > // orders/fixtures/test_data.json > [ > { > "model": "orders.order", > "pk": 1, > "fields": { > "account": 1, > "number": 1, > "bill_address": 1 > } > }, > { > "model": "orders.orderaddress", > "pk": 1, > "fields": { > "account": 1, > "order": 1, > "country": "US", > } > } > ] > > > (an Account with the primary key of 1 already exists at the time of > ``loaddata``) > > > The error I get with `python manage.py loaddata test_data orders` is: > > django.db.utils.IntegrityError: Problem installing fixture > '/opt/myproject/apps/orders/fixtures/test_data.json': Could not load > orders.Order(pk=1): insert or update on table "orders_order" violates foreign > key constraint "bill_address_id_refs_id_3a4d3fef" > DETAIL: Key (bill_address_id)=(1) is not present in table > "orders_orderaddress". > > > The above fixtures load locally, and they load during test running (with > Postgres) a number of times, but for some reason I get that error when using > `manage.py loaddata ...`. > > > On Sunday, June 16, 2013 7:40:02 PM UTC-4, Russell Keith-Magee wrote: > > Circular dependencies *shouldn't* be a problem on PostgreSQL because all > constraints are set DEFERABLE INITIALLY DEFERRED; that means no constrain > checks should be performed are performed until the transaction boundary, so > all circular references shouldn't be a problem. > > Ticket #3615 exists because MySQL's implementation of DEFERABLE INITIALLY > DEFERRED under InnoDB is, to use the technical term, "Broken". It's unrelated > to any problem you may have found in PostgreSQL, because PostgreSQL gets the > underlying behaviour right. > > Beyond that, we need a specific test case to take this any further. As it > stands, I'm not aware of any problems loading fixtures into PostgreSQL. If > you are able to construct and provide a set of models (which you have done) > and simple fixture (which you haven't) that fails reliably, we have a new bug > on our hands, and you should open a ticket with all the details you can > provide. Confirming whether this is a problem with the alpha, or an ongoing > problem would also be helpful. > > Yours, > Russ Magee %-) > > On Mon, Jun 17, 2013 at 6:22 AM, Yo-Yo Ma <[email protected]> wrote: > There doesn't appear to be a way to load fixtures from JSON (using Postgres - > works fine in sqlite3) for the following models: > > > class Collection(models.Model): > main_thing = models.OneToOneField( > 'things.Thing', > null=True, > on_delete=models.SET_NULL > ) > > class Thing(models.Model): > collection = models.ForeignKey( > 'collections.Collection' > ) > > > Here is the exception: > > Problem installing fixture 'my_fixture.json': Could not load > collections.Collection(pk=1): insert or update on table > "collections_collection" violates foreign key constraint > "main_thing_id_refs_id_3a4d3fef" > DETAIL: Key (main_thing_id)=(1) is not present in table "things_thing". > > I'm not sure if the issue is due to the unique constraint implied by a > OneToOneField, or if it's just related to this issue: > https://code.djangoproject.com/ticket/3615 (seems like that ticket and > related ones have been closed for years, so possibly not related). > > Any thoughts? > > Note: I'm using @1.6a1 > > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/django-developers. > For more options, visit https://groups.google.com/groups/opt_out. > > > > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/django-developers. > For more options, visit https://groups.google.com/groups/opt_out. > > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
