Google was 100% unhelpful when I was beating my head against this problem but it turned out to be relatively simple to resolve.
My linux debugging skills are a little rusty but eventually i looked up the syntax to have strace dump to a text file and then it was pretty obvious.
When spyserver receives a connection and is configured for an rtl-sdr device, it beats the bushes looking for librtlsdr.so
Naturally you have to have that library installed, and it has to have been built for the same architecture as spyserver was built for, and that should be covered by having installed the rtl-sdr packages on your linux system.
The trouble is that it's installed as librtlsdr.so.0
So in my situation, running on armbian on a 64 bit arm with the armhf version of librtlsdr installed, I did this:
sudo ln -s /usr/lib/arm-linux-gnueabihf/librtlsdr.so.0 /usr/lib/arm-linux-gnueabihf/librtlsdr.so
The exact path you need to specify depends on your architecture.
If spyserver were compiled like a normal dynamic elf binary, you could have used "ldd" to see what it's trying to load so that you can then supply it, but it isn't (ldd simply says it's not a dynamic binary).
If you do "strace -f -o spyserver.trace ./spyserver" and then attempt to connect, the logs recorded in spyserver.trace will reveal where it tried to find librtlsdr.so and didn't find it.