Hi Peter, Before you start it off again can you describe how you are doing the import; ie time period per import, one big import, multiple smaller periods? How is you data organised on disc, one large directory with all file, year by year directories etc.
Gary On Tuesday, 16 February 2021 at 00:00:27 UTC+10 peter.su...@gmail.com wrote: > Hi Gary, > > today my Raspberry 4 arrived, which will hopefully be my long-running > weather computer. > I tried to import all my data on Pi3. It worked back for about 8 years, > but then the performance crashed into hell. > The import of one year took about 1 week and was getting worse. > > Now I'm starting it all over on a fresh install and report, what will be > the result. > > Peter > gjr80 schrieb am Montag, 1. Februar 2021 um 23:28:04 UTC+1: > >> Hi Peter, >> >> On Tuesday, 2 February 2021 at 06:03:36 UTC+10 peter.su...@gmail.com >> wrote: >> >>> Hey Gary, >>> its looking perfect. I just separated the logs for each year and right >>> now I am testing the 6th year in a row, doing a quick dry-run. >>> The script is finding some duplicates, but it looks like every complete >>> year from 01/01 to 12/31 should be imported into the database. >>> >> >> That is interesting, the screen output you posted originally clearly >> showed an error decoding a date-time, I am surprised that re-arranging the >> log files fixed that. The first person that trialled the WD import module >> was in a similar boat to you; they had many years of WD data and when they >> first tried to import the data they did it in one run and not only did it >> take many hours but wee_import eventually ground to a halt and crashed >> (the exact details escape me). So importing/processing a year at a time may >> be more manageable and safer. >> >> The duplicates issue is not uncommon, the few suites of WD logs I have >> worked on all contained the odd duplicate date-time. wee_import will >> simply ignore the duplicates, the only issue is when the data for each >> duplicate is different. Probably not going to make too much difference over >> 15 years of data at one minute intervals. >> >> One more question from my side: Would you prefer a MySQL database with >>> all this mass of data? >>> >> >> I think the consensus is that unless you have good reason to use >> MySQL/MariaDB stick with SQLite. SQLite is faster and simpler to >> backup/restore. SQLite will easy handle all of your 15 years of data. >> >> Gary >> > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c9b2b67d-9377-488f-8d5c-94662cf13315n%40googlegroups.com.
