gravitystorm left a comment (openstreetmap/openstreetmap-website#5667)
In my own projects, we use a different approach for data migrations.
* First, we avoid using rails built-in database migrations. These are great for
structure changes, but not for data changes, since they can't be tested. And
there's often some edge cases that we want to have confidence about before we
run the data migration and tests are the foundation for confidence.
* We create a job to migrate the data, so that it runs in a background process.
We have a rake task that simply queues the job and nothing else. This is
because rake tasks are convenient for us to run, but again are very hard to
test. Jobs are easy to test.
* We have a separate queue for slow jobs, with a separate set of workers, so
that if it takes e.g. 10min to run we are still sending out confirmation emails
in the meantime.
* For the task itself, we write it in a way that it can be retried multiple
times. So for example we only select the objects that have not yet been
migrated (e.g. `Note.where(description: nil).in_batches`) so that if the job is
retried, it starts from where it left off, not from the start again.
We do this because in so many cases either we forgot about an edge case but
picked it up in testing, or there turns out to be some unexpected data that
messes things up (an example would be a note with no comments, even if there's
no way that could have happened, in theory, somehow it always does). It also
avoid any worries about locks, long-running transactions, transactions with
large amounts of uncommitted changes, timeouts, network interruptions, db
process restarts and so on.
I say all this to provide an alternative approach that I know works. But in
this case, since @tomhughes will be running the actual migration process,
whatever approach he prefers is what we should do!
--
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/pull/5667#issuecomment-2659898440
You are receiving this because you are subscribed to this thread.
Message ID:
<openstreetmap/openstreetmap-website/pull/5667/c2659898...@github.com>
_______________________________________________
rails-dev mailing list
rails-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/rails-dev
