Re: New to WSPR Daemon


Gwyn Griffiths
 
Edited

Hello Greg
Thanks for the additional explanations and details. They give a useful picture of where you are with csv file data and the approaches that you take. It is clear that once past the first stage of getting the data columns you want the subsequent steps from wsprnet csv or wsprdaemon TimescaleDB files will be the same. As fields such as numerical lat and lon for tx and rx are already in WsprDaemon this may reduce the load at the analysis steps - that was our hope.

I am in no doubt that multithread (and cluster) approaches are needed with this data. WsprDaemon already has 390 million spots online (from July 2020 onward) and this can, and does, result in slow response to a number of queries and hence Grafana graphics. For now, in the scheme of things, being able to see a plot of spot count per hour of day for each day over six months is a few tens of seconds is still useful, and a marvel.

But, these 390 million spots are only taking up 138 GB of the 7TB disk space Rob has made available - so different approaches such as those you describe are going to be needed to look at data over a whole sunspot cycle that the WsprDaemon should be able to hold.

Thanks for permission to abstract from your posts on this topic for our TimescaleDB guide.

73
Gwyn G3ZIL

Join wsprdaemon@groups.io to automatically receive all group messages.