On Saturday, January 28, 2017 at 2:06:11 PM UTC-5, Mike Orr wrote:
>
>
> How are people synchronizing something transactional with something
> non-transactional, especially a database record with a filesystem
> file.
>
I deal with this a lot in the area of S3 file archiving.
I abort the transaction if the s3 upload fails. Usually I can delete the
file via a except/finally clause -- but there are still edge cases where I
could end up with a stray because the cleanup activity can fail.
To handle that, I use a second SqlAlchemy connection in autocommit mode for
bookkeeping, then run periodic tasks to reconcile issues later on.
from memory, it looks something like this (psuedocde!)
try:
# transactional record
objectRecord = {"id":id, "filename": filename}
dbSessionTransaction.add(objectRecord)
dbSessionTransaction.flush()
# autocommit log
objectLog = {"id": id, "status": "prep"}
dbSessionAutocommit.add(objectLog)
dbSessionAutocommit.flush()
# mark a partial upload
objectLog.status = 'uploading'
dbSessionAutocommit.flush()
boto.upload_file(filename)
# mark uploaded
objectLog.status = 'uploaded'
dbSessionAutocommit.flush()
# mark the file
objectRecord.status = 'uploaded'
objectRecord.timestamp_upload = datetime()
dbSessionTransaction.commit()
except:
# mark the autocommit as "deletion-start"
# try to delete the file
# mark the autocommit as "deleted"
I periodically look through the autocommit logs to see if anything is stuck
and make sure deleted items are not in s3.
it's definitely overkill, but it has so far caught all the (rare) edge
cases.
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/00c9a773-ac1b-40c7-9198-d46132f4c17e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
