> Den 12/08/2015 kl. 04.47 skrev yakkades...@gmail.com:
>
> for row in rows:
> dp = DataPoint.objects.get(Taken_datetime=row['date'])
>
> sensorToAdd = []
> for sensor in sensors:
> s = Sensor.objects.get(Name=sensor.name, Value=sensor.value )
> sensorToAdd.append( s )
>
> dp.Sensors.add( sensorToAdd )
>
> In the actually app I bulk create all the DataPoint and Sensor instances.
> The problem is that |dp.Sensors.add( sensorToAdd )| does a lot of hits on the
> db. I want a way bulk add all the sensors.
Try fetching all the data you need from the database up-front and place objects
in a dict. Depending on the volume of your data, you may want to split this
into reasonable batches:
from django.db.models import Q
datapoints = DataPoint.objects.filter(Taken_datetime__in={r['date'] for r in
rows})
datapoints_map = {(d.Taken_datetime, d) for d in datapoints}
# To generate efficient SQL, make sure (name, value) pairs are unique
unique_sensor_values = {(s.name, s.value) for s in my_list_of_sensors}
sensors_q = Q()
for name, value in my_unique_sensor_values:
sensors |= Q(Name=name, Value=value)
sensors = Sensor.objects.filter(sensors_q)
sensors_map = {((s.name, s.value), s) for s in sensors}
This reduces your queries to only two. You can then bulk-insert m2m relations
per-object like this:
for some_date, some_sensors in my_data:
dp = datapoints_map[some_date]
dp.Sensors.add(*[sensors_map[(s.name, s.value)] for s in some_sensors])
If you need better bulk-insert performance than this, you can convert the m2m
relation between Sensor and DataPoint to an explicit m2m model. You can then
bulk-insert all m2m relations in one go instead of per-object.
You should probably add indexes on DataPoint.Taken_datetime and Sensors.[Name,
Value] to increase query performance.
Erik
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/7FCE3968-7C15-4D92-8DB3-E4A6EE56442B%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.
