Gave it up now.
Pulled out my old Raspi2, there it works without any problems. Spyserver generates between 40% and 60% load @1000MHz, even at 32bit float format, when using full IQ. With highest decimation, load is 80-85%  but no stuttering at all.
So, I'm fine now with the Raspi2.

Thanks again for your support!

Eckhard