I added a third IF at 110MHz. Now performance is acceptable.

Two examples:

Wide scan: 200MHz output of a SI5351 through 50dB attenuator to see the harmonics. The actual maximum scan frequency is 1890MHz


The spurs at 25MHz and related harmonics are probably from the 25MHz reference clock on the ADF4351 module. Limited mixer spurs.

Small scan: Measuring a 1 MHz sinus signal at calibrated level of -80dBm



The power scale is 10dB off, actual noise floor is thus at -105dBm. Max input signal is -30dBm (actual -20dBm) so dynamic range is not very good (85dB, I was hoping for more then 90dB) but low frequency performance is usable and the zero hertz response small enough.
Now I have to put everything in a nice box, not my biggest competence