Hi Peter, On Tuesday, 2 February 2021 at 06:03:36 UTC+10 peter.su...@gmail.com wrote:
> Hey Gary, > its looking perfect. I just separated the logs for each year and right now > I am testing the 6th year in a row, doing a quick dry-run. > The script is finding some duplicates, but it looks like every complete > year from 01/01 to 12/31 should be imported into the database. > That is interesting, the screen output you posted originally clearly showed an error decoding a date-time, I am surprised that re-arranging the log files fixed that. The first person that trialled the WD import module was in a similar boat to you; they had many years of WD data and when they first tried to import the data they did it in one run and not only did it take many hours but wee_import eventually ground to a halt and crashed (the exact details escape me). So importing/processing a year at a time may be more manageable and safer. The duplicates issue is not uncommon, the few suites of WD logs I have worked on all contained the odd duplicate date-time. wee_import will simply ignore the duplicates, the only issue is when the data for each duplicate is different. Probably not going to make too much difference over 15 years of data at one minute intervals. One more question from my side: Would you prefer a MySQL database with all > this mass of data? > I think the consensus is that unless you have good reason to use MySQL/MariaDB stick with SQLite. SQLite is faster and simpler to backup/restore. SQLite will easy handle all of your 15 years of data. Gary -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b4d2b792-3cbb-4fb5-9964-d3b5fbdcca0en%40googlegroups.com.
