Dec. 11, 2019, 2:24 p.m.
This problem was met during last update of our code database for one e-commerce project. We have Cart model with OrderEntry relationship. Cart was used to connect specific purchases with specific Users. OrderEntry was used to store data about Items purchased by User.
Let’s say User with id=1 is buying items: Canon EOS 6D camera (Item.id = 5), Canon 50mm f2 lenses (Item.id = 8). We will have Cart object with creation date = now, User = current user and 2 OrderEntry objects related to Items with id=5 and id=8. Nothing interesting here. Could be built better or worse.
During last few weeks we were working hard on big changes to Item model. Actually changes were so big we decided to create separate model ItemVariation2 and copy existing data there with keeping old IDs. Such decision was made in order to check everything on production before making switches.
So now we have ItemVariation2 model which has exactly the same IDs that Item model had. It means that all old OrderEntries could be switched from Item to ItemVariation2 and no information will be lost. How to do it?
Before changes we have this:
item = models.ForeignKey(
Item,
null=True,
on_delete=models.SET_NULL,
related_name="items")After changes we have this:
item = models.ForeignKey(
ItemVariation2,
null=True,
on_delete=models.SET_NULL,
related_name="items")Than you do makemigrations command and migrate. Everything works okay, no errors. You do runserver, pages are opened, admin works ok. If you have tests in your code (you have, don’t you? ;) ) they’ll most probably pass without problems.
You’re creating new ItemVariation2, it gets ID that wasn’t present in old Item model. You create new OrderEntry with item.id = ID of this new ItemVariation2. And BOOM! — you’ve got error! Your new id is breaking constraint of field because there is no such id in Item model. But what a hell? You changed your model and did migration, why does it keep connection to old one?
The reason is that you have constraint in your database. In order to check it you can start dbshell management command and do the following:
shop_db=> \d+ cart_orderentry
Table "public.cart_orderentry"
Column | Type | Modifiers | Storage | Stats target | Description
----------+---------+--------------------------------------------------------------+---------+--------------+-------------
id | integer | not null default nextval('cart_orderentry_id_seq'::regclass) | plain | |
price | integer | not null | plain | |
quantity | integer | not null | plain | |
cart_id | integer | not null | plain | |
item_id | integer | | plain | |
Indexes:
"cart_orderentry_pkey" PRIMARY KEY, btree (id)
"cart_orderentry_82bfda79" btree (item_id)
"cart_orderentry_c44d83f7" btree (cart_id)
Foreign-key constraints:
"cart_orderentry_cart_id_650e0a99_fk_cart_cart_id" FOREIGN KEY (cart_id) REFERENCES cart_cart(id) DEFERRABLE INITIALLY DEFERRED
"cart_orderentry_item_id_73f381f4_fk_catalogue_item_id" FOREIGN KEY (item_id) REFERENCES catalogue_item(id) DEFERRABLE INITIALLY DEFERREDThe last line is the reason of problem. What it does is actually checking the list of existing IDs for Item model and raises error if you try to create relationship with non-existent object. We did create new object ItemVariation2 with new ID but our database is checking Item model instead.
In order to fix it you can go a bit hacky way and manipulate your database directly:
shop_db=> ALTER TABLE cart_orderentry
shop_db-> DROP CONSTRAINT cart_orderentry_item_id_73f381f4_fk_catalogue_item_id,
shop_db-> ADD CONSTRAINT cart_orderentry_item_id_73f381f4_fk_catalogue_new_itemvariation2_id FOREIGN KEY (item_id) REFERENCES catalogue_new_itemvariation2(id) DEFERRABLE INITIALLY DEFERRED;
NOTICE: identifier "cart_orderentry_item_id_73f381f4_fk_catalogue_new_itemvariation2_id" will be truncated to "cart_orderentry_item_id_73f381f4_fk_catalogue_new_itemvariation"Check it again:
shop_db=> \d+ cart_orderentry
Table "public.cart_orderentry"
Column | Type | Modifiers | Storage | Stats target | Description
----------+---------+--------------------------------------------------------------+---------+--------------+-------------
id | integer | not null default nextval('cart_orderentry_id_seq'::regclass) | plain | |
price | integer | not null | plain | |
quantity | integer | not null | plain | |
cart_id | integer | not null | plain | |
item_id | integer | | plain | |
Indexes:
"cart_orderentry_pkey" PRIMARY KEY, btree (id)
"cart_orderentry_82bfda79" btree (item_id)
"cart_orderentry_c44d83f7" btree (cart_id)
Foreign-key constraints:
"cart_orderentry_cart_id_650e0a99_fk_cart_cart_id" FOREIGN KEY (cart_id) REFERENCES cart_cart(id) DEFERRABLE INITIALLY DEFERRED
"cart_orderentry_item_id_73f381f4_fk_catalogue_new_itemvariation" FOREIGN KEY (item_id) REFERENCES catalogue_new_itemvariation2(id) DEFERRABLE INITIALLY DEFERREDAfter that you’re able to add new OrderEntry => ItemVariation2 relationships with new ID without any problems. But you’re in deep shit if you add new field somewhere, do makemigrations and migrate than. You’ll get the following error:
ValueError: Unhandled pending operations for models: <your_app>.<your_model> (referred to by fields: <related_field>)It’s good if you get this situation soon but it can take few days/weeks/months before you do your new migrations and get the error. The only way to fix it is to revert migrations and most probably lost your data.
For me it seemed strange that we needed to do direct database manipulations having so powerful migration mechanism built-in Django. There should be better way. And actually it is. And it is so freaking simple:
# -*- coding: utf-8 -*-
# Generated by Django 1.9.4 on 2017-08-08 14:20
from __future__ import unicode_literals
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('cart', '0011_cart_admin_comment'),
]
operations = [
migrations.AlterField(
model_name='orderentry',
name='item',
field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='items', to='catalogue_new.ItemVariation2'),
),
]Everything looks ok here. But after such migration you’ll have constraint in DB kept for old relationships. That happens because migrations mechanism of Django checks all the previous migrations and do database manipulations based on all of them. We already have field named item in previous migrations. It depends on other migrations actually. But Django’s makemigrations command doesn’t change dependencies if you alter foreignkey field :( So before doing migrate command just change your migration file:
# -*- coding: utf-8 -*-
# Generated by Django 1.9.4 on 2017-08-08 14:20
from __future__ import unicode_literals
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('catalogue_new', '0001_initial'), # this one is added by me
('cart', '0011_cart_admin_comment'),
]
operations = [
migrations.AlterField(
model_name='orderentry',
name='item',
field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='items', to='catalogue_new.ItemVariation2'),
),
]Here I added external dependency needed to keep constraint for database. Just put there name of application which your field related to, and name of migration where this field is described. My ItemVariation2 model lives in catalogue_new app, it was created during initial migration so I added it here. Do migrate command and check constraint in dbshell now:
shop_db=> \d+ cart_orderentry
Table "public.cart_orderentry"
Column | Type | Modifiers | Storage | Stats target | Description
----------+---------+--------------------------------------------------------------+---------+--------------+-------------
id | integer | not null default nextval('cart_orderentry_id_seq'::regclass) | plain | |
price | integer | not null | plain | |
quantity | integer | not null | plain | |
cart_id | integer | not null | plain | |
item_id | integer | | plain | |
Indexes:
"cart_orderentry_pkey" PRIMARY KEY, btree (id)
"cart_orderentry_c44d83f7" btree (cart_id)
"cart_orderentry_item_id_73f381f4_uniq" btree (item_id)
Foreign-key constraints:
"cart_ordere_item_id_73f381f4_fk_catalogue_new_itemvariation2_id" FOREIGN KEY (item_id) REFERENCES catalogue_new_itemvariation2(id) DEFERRABLE INITIALLY DEFERRED
"cart_orderentry_cart_id_650e0a99_fk_cart_cart_id" FOREIGN KEY (cart_id) REFERENCES cart_cart(id) DEFERRABLE INITIALLY DEFERREDNow db constraints are changed just from Django without touching database from shell. Migration history is kept consistent. You can add new fields to any application without any problems.
P.S.: Probably it’s well described somewhere in documentation and is obvious for every Django developer but it took us few hours to fix it and solution was found in absolutely not related docs.