not much advice but number of questions instead besides saying that your posts are really interesting to follow.
1. Why not lower cutoff frequency on the first LPF somewhere around 1.3GHz?
2. What are you using as 10dB gain blocks, SPF5189Z modules?
3. Which version of the ADL5801 mixer are you using? With or without input baluns?
4. Typo, or why have you changed from AD603 to AD605 in the log detector?