Re: [weewx-user] Independent weewx backup script

Way back in 2014 (wow) when we had that long thread about this, many of us checked our hundreds of backups with 'pragma integrity_check' and never found any that were not restorable, so I'm pretty ok with taking that risk still. On Monday, January 11, 2021 at 5:26:08 PM UTC-8 Tom Keffer wrote:

Re: [weewx-user] Independent weewx backup script

i use ‘cp’. heck, i use ‘scp’… i just make sure i work in the window from 2 mins after archive interval (long after my archive updates are completed) to a few secs before next archive interval. could make BACKUP a new built-in report like FTP and RSYNC - unless want to make backup independent of

Re: [weewx-user] Independent weewx backup script

I actually think 'cp' is pretty safe. When WeeWX writes a record to the database, it has to update the main archive, as well as many daily summaries. To make sure this is all done atomicallly, it does it as one big transaction. So, unless your 'cp' has the unfortunate timing of happening at the ex

Re: [weewx-user] Independent weewx backup script

For for 1,357,184 records in the db records in my 340 MB database's archive table Using Tom's variant from a local file to a backup file in the same working directory Dockstar = 46.8 secs via python, 35.1 secs to cp (usb2 laptop drive) pi3 = 44.3 secs via python, 40.

Re: [weewx-user] Independent weewx backup script

The program can be simplified even more: import sqlite3 with sqlite3.connect('/home/weewx/archive/weewx.sdb') as original: with sqlite3.connect('/home/weewx/archive/weewx.sdb.backup') as backup: original.backup(backup, pages=10) I'm finding that the time it takes to do the backup dep

Re: [weewx-user] Independent weewx backup script

as a once-off i did nothing fancy, even hard-coding the number of pages and filename of database (trivial to fiddle), using shell time to measure elapsed time import sqlite3 con = sqlite3.connect('/opt/weewx-4.2.0-test/archive/weewx.sdb') bck = sqlite3.connect('backup.db') with bck: con.bac

Re: [weewx-user] Independent weewx backup script

On Monday, January 11, 2021 at 5:03:14 AM UTC-8 graha...@gmail.com wrote: > conclusion: only seriously under-powered boxes would be unable to complete > within typical 300 sec archive interval. > would be good if someone with such a box gave it a try > > If you can point us at the exact script yo

Re: [weewx-user] Independent weewx backup script

looking at https://sqlite.org/backup.html (extract below; my emphasis), the Backup API restarts the backup if an update (not a read) occurs during the backup → might silently never complete if backup takes longer than archive interval. this could be dealt with by

Re: [weewx-user] Independent weewx backup script

This comes up very frequently as new weewx users come onboard. Take a look back at the weewx-users archives for lots of previous (excellent) discussions for how to automate backing up your db and 'verifying' that the backup is restorable. There's an old sysadmin credo saying that if you haven

Re: [weewx-user] Independent weewx backup script

Good point. Doing all this without stopping weewx is much nicer. In fact, after a year of data (in my case), the sqlite3 approach will not take that long. I will try that as well. I see that data backup seems to be a hot topic... tke...@gmail.com schrieb am Sonntag, 10. Januar 2021 um 14:39:13

Re: [weewx-user] Independent weewx backup script

I use mysql and keep a week of files which are backed up to another server. I use the name of the day of the week and that way it overwrites the old file with the same name and you get a rolling 7 files. #keep a weeks worth of weewx backups /usr/bin/mysqldump --add-drop-table --user=root --passw

Re: [weewx-user] Independent weewx backup script

On Sun, Jan 10, 2021 at 9:06 AM Timothy L wrote: > For my own personal understanding as a newcomer I would like to ask how is > it possible to lose a data record using a logger such as in the Vantage Pro > 2 series that should report once weewx is restarted after the backup? > Wouldn't the logger

Re: [weewx-user] Independent weewx backup script

Trying it. Seems very slow compared to a file copy. Doing it one page at a time is really slow. Like 100x slower than a file copy. I don't have any sense as to the page size of this database. I'm probably going to end up doing 100 pages at a time. The example here https://docs.python.org/3/li

Re: [weewx-user] Independent weewx backup script

For my own personal understanding as a newcomer I would like to ask how is it possible to lose a data record using a logger such as in the Vantage Pro 2 series that should report once weewx is restarted after the backup? Wouldn't the logger have recorded the weather event and then transferred that

Re: [weewx-user] Independent weewx backup script

wee_database --backup leveraging the data bindings in weewx.conf sounds like a beneficial core utility service that would then be called from a script/cron. On Sunday, January 10, 2021 at 10:22:43 AM UTC-5 tke...@gmail.com wrote: > If the backup takes long enough, it could interfere with w

Re: [weewx-user] Independent weewx backup script

If the backup takes long enough, it could interfere with writing a record to the database. Eventually, the write will time out, causing weewxd to restart from the top. It won't crash weewxd (that is, cause it to exit), nor corrupt the database, but the record would be lost. That's the advantage of

Re: [weewx-user] Independent weewx backup script

I understand database consistency in a transactional database and I'm wondering about the risk of copying weewx.sdb without stopping weewx first. Would you possibly lose an in-flight transaction or might the entire sdb be inconsistent and unusable? An in-flight copy would seem to be similar to

Re: [weewx-user] Independent weewx backup script

I have a script that stops weewx, makes a local copy of the db restarts weewx and then copies the backup to google drive. The copy takes less than 2 minutes so I don't lose data. On Sun, Jan 10, 2021, 5:39 AM Tom Keffer wrote: > Your approach will certainly work, but requires stopping weewxd fo

Re: [weewx-user] Independent weewx backup script

Your approach will certainly work, but requires stopping weewxd for what could potentially be a long period of time, so you might miss a weather event. Another approach is to use the sqlite3 ".backup" command. Replace your tar command with tar czf $dest/$archive_file $backup_files2 $backup_files3

[weewx-user] Independent weewx backup script

For some time, I was looking for an easy and independent (from weewx) way to automatically backup my weewx data, as I do not want to lose data if the Micro SD breaks down. Recently, I found this small repo on github which only contains a backup scri