On 2019-12-02 16:49, Michael Torrie wrote: > On 12/1/19 7:50 PM, Tim Chase wrote: > > After sparring with it a while, I tweaked the existing job so > > that it chunked things into dbm-appropriate sizes to limp > > through; for the subsequent job (where I would have used dbm > > again) I went ahead and switched to sqlite and had no further > > issues. > > How did you replace a key/value store with a relational database? > Is a SQLite database fast enough at this sort of thing that it > wasn't really designed for?
It was certainly slower, though it wasn't so bad once I had proper indexing and submitted queries that pulled back multiple results in one query. But even with the slightly slower run-time aspects, it was still faster than starting a job (expecting it to run to completion overnight), having it crash, manually deleting my cache, and manually resuming from where it left off, all multiple times. And all said, since it was network I/O bound, once I had the populated cache (resulting cache.db file was about 1TB...thank goodness for transparent compression with ZFS), turnarounds took more like 30min rather than 3 days. More the "go work on something else and come back" than the "let it run overnight". -tkc -- https://mail.python.org/mailman/listinfo/python-list
