Re: New to WSPR Daemon


Greg Beam <ki7mt01@...>
 
Edited

Hi Gwyn,

I've been slammed with web / domain transfer work the last few weeks and haven't had much time for radio related activity, though I did manage to splash out for a new rig: Elecraft => K4HD

Thanks for the additional links to your project objectives as that clarified a number of things.

Regarding PySpark - I used that because it was / is easy to get up and running. However, the real power / speed with Spark (IMHO) comes in with Scala. It's much faster as it's compiled rather than interrupted at runtime. It's also, like Java, Type-Safe, so there's little-to-no concern about mangling data types during one's process steps.

Pairing down (map reduce) larger data sets into fact tables is where I'll probably focus most of my efforts as that allows any number of rendering / graphing engines to easily consume the output. Getting the data into a usable structure is my first goal. After thinking on this more, I'll probably create an install-able Linux package that wraps each Scala Jar File Assembly with a bash script so it's easier for end users to call. This is what I'm pondering, but, I've not set anything into motion yet.

Re ClickHouse -  looks very interesting (and fast). That may be the fastest solution I've seen to date for consuming large chunks of data in a columnar format. I glanced through the docs, it looks to be very powerful indeed and definitely warrants a more thorough read.

73's
Greg, KI7MT

Join wsprdaemon@groups.io to automatically receive all group messages.