Hi Gary, today my Raspberry 4 arrived, which will hopefully be my long-running weather computer. I tried to import all my data on Pi3. It worked back for about 8 years, but then the performance crashed into hell. The import of one year took about 1 week and was getting worse.
Now I'm starting it all over on a fresh install and report, what will be the result. Peter gjr80 schrieb am Montag, 1. Februar 2021 um 23:28:04 UTC+1: > Hi Peter, > > On Tuesday, 2 February 2021 at 06:03:36 UTC+10 peter.su...@gmail.com > wrote: > >> Hey Gary, >> its looking perfect. I just separated the logs for each year and right >> now I am testing the 6th year in a row, doing a quick dry-run. >> The script is finding some duplicates, but it looks like every complete >> year from 01/01 to 12/31 should be imported into the database. >> > > That is interesting, the screen output you posted originally clearly > showed an error decoding a date-time, I am surprised that re-arranging the > log files fixed that. The first person that trialled the WD import module > was in a similar boat to you; they had many years of WD data and when they > first tried to import the data they did it in one run and not only did it > take many hours but wee_import eventually ground to a halt and crashed > (the exact details escape me). So importing/processing a year at a time may > be more manageable and safer. > > The duplicates issue is not uncommon, the few suites of WD logs I have > worked on all contained the odd duplicate date-time. wee_import will > simply ignore the duplicates, the only issue is when the data for each > duplicate is different. Probably not going to make too much difference over > 15 years of data at one minute intervals. > > One more question from my side: Would you prefer a MySQL database with all >> this mass of data? >> > > I think the consensus is that unless you have good reason to use > MySQL/MariaDB stick with SQLite. SQLite is faster and simpler to > backup/restore. SQLite will easy handle all of your 15 years of data. > > Gary > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d6822694-7e91-42a2-b2d4-b661bf376b49n%40googlegroups.com.
