Thanks - you've definitely given me some stuff to think about. I'm doing
XHR requests - returning JSON for the scraping (but probably later will
have normal pages so I will definitely look at your Etag suggestion - I'm
not familiar with that so will look into it).
Given it's XHR and JSON I pr
If you are web scraping you really need your code to be as efficient as
possible and to do as little as possible. Firstly, make sure you are using
everything the servers of the websites you are scraping are giving you to
decide whether to bother downloading the page. For example, check the etag
and
Hi Dan,
Thanks for the suggestion, it's a web scraper (run as a django management
command) which then saves the data to the database via the Django ORM.
Given it's a scraper rather than a form (or view) is the above suggested
function an ok way to proceed or would you suggest something else is
Hi Yunti,
You could go up a level in the structure of your application and apply the
logic there, where there is more support.
Are you using Django forms? The ModelForm class pretty much does what you
want, it examines form data, validating it against its type and any
validation rules you have s
Jani,
Thanks for your reply - you explained it much more concisely than I did. :)
Good to have it confirmed that update_or_create() doesn't quite do what I
needed - I was confused as to whether it would or not.
Thanks for taking the time to do that function, that looks ideal. I'll test
it out.
Your problem lies on the way Django actually carries out create or update.
As name suggest, create or update does either one. But that's what you
don't want - you want conditional update.
Only update if certain fields have been changed. Well this can be done
few ways.
So you want to do
"up
Carsten ,
Thanks for your reply,
A note about the last statement: If a Supplier object has the same
unique_id, and all
other fields (in `defaults`) are the same as well, logically there is no
difference
between updating and not updating – the result is the same.
The entry in the database is
Carsten ,
Thanks for your reply,
A note about the last statement: If a Supplier object has the same
unique_id, and all
other fields (in `defaults`) are the same as well, logically there is no
difference
between updating and not updating – the result is the same.
The entry in the database is
Hi Yunti,
Am 05.11.2015 um 18:19 schrieb Yunti:
I have tried to use the update_or_create() method assuming that it would
either, create
a new entry in the db if it found none or update an existing one if it found
one and had
differences to the defaults passed in - or wouldn't update if there
I have tried to use the update_or_create() method assuming that it would
either, create a new entry in the db if it found none or update an existing
one if it found one and had differences to the defaults passed in - or
wouldn't update if there was no difference. However it just seemed to
rec
10 matches
Mail list logo
