Re: WSPR data analysis #wspr #qcx40


Ted :

The simple approach that I have used to deal with these massive WSPR .csv files is to pre-process the file on a Linux machine
to extract the relevant data into a new file, discarding all of the records that I don't care about. Then I transfer that  new file to my
Windows machine and then open it with Excel.

The grep utility on Linux/Unix is line-based so it reads in one line at a time and processes that very efficiently. It can search using any 
regular expression so it is a simple matter to grep the .csv searching for your callsign and then use ">" to redirect the output
to a new text file.   Something like :  grep "VE3WMB"  wspr.csv  > newcsvfile.csv    would search wspr.csv for VE3WMB and output
only those lines. The '>' takes those output lines and redirects them to a new file named newcsvfile.csv . 

You can do this on any Linux/Unix box, even a Raspberry Pi.  If you don't have a Linux machine you could create a USB stick
with a live Linux distribution and boot from that without installing Linux on your computer. Or if this seems to difficult find someone
you know who is a Linux user and get them to pre-process the file for you.  

The advantage of this approach is that you separate the "wheat from the chaff" before you try to do anything with the data.  
Using a database you are still loading in a pile of information that you really just want to discard.


Michael VE3WMB 

Join to automatically receive all group messages.